Loading...
Build your personal recon pipeline — scan while you sleep, get alerts when something critical is found
Chain tools together so your recon runs while you sleep
A recon pipeline chains multiple tools together so the output of one feeds into the input of the next. Instead of running subfinder, then httpx, then nuclei manually, you build a script that does all of it in one command. This chapter teaches you to build, test, and deploy your first automated recon pipeline — from a simple one-liner to a full multi-stage system.
Every ProjectDiscovery tool follows the same pattern: stdin → tool → stdout. This means you can connect them infinitely. Learn the pipe basics first — once you understand this pattern, you can build any pipeline by just swapping tools in and out.
subfinder -d example.com -silent | httpx -silent -title -status-code | head -20The simplest pipeline: subdomains → live probe → preview. Three tools, one command, no temp files.
cat domains.txt | subfinder -silent | dnsx -silent -a -resp-only | sort -u > ips.txtDomain to IP pipeline: batch subdomain enumeration → DNS resolution → unique IP list
subfinder -d example.com -silent | httpx -silent -title -status-code -tech-detect | grep -v 404 | tee live-sites.txtFull recon one-liner: enumerate → probe → filter 404s → display AND save output simultaneously
subfinder -d example.com -silent | dnsx -silent -a -resp-only | naabu -list - -top-ports 1000 -silent | tee port-scan.txtComplete surface scan: subdomains → IPs → port scan — all without writing a single temp file
subfinder -d example.com -silent | httpx -silent | nuclei -silent -t ~/nuclei-templates/ -o nuclei-results.txtThe golden pipeline: find subs → probe live → scan for vulns — the foundation of every automated workflow
cat urls.txt | grep -E '\.js$' | nuclei -silent -tags exposures -o js-secrets.txtJavaScript scanning pipeline: filter JS files from URL list → scan for API keys and secrets with Nuclei
TIPS
Move from one-liners to a proper bash script. A script gives you control over input validation, error handling, output organization, and timestamped logging. Start simple — this script takes a domain, enumerates subdomains, probes them, and scans for vulnerabilities with organized output.
#!/bin/bash
TARGET=$1
OUTDIR="recon-$TARGET-$(date +%Y%m%d-%H%M)"
mkdir -p $OUTDIR
subfinder -d $TARGET -silent -o $OUTDIR/subs.txt
dnsx -l $OUTDIR/subs.txt -silent -a -resp-only -o $OUTDIR/ips.txt
httpx -l $OUTDIR/subs.txt -silent -title -status-code -tech-detect -o $OUTDIR/live.txt
nuclei -l $OUTDIR/live.txt -silent -o $OUTDIR/nuclei-results.txt
echo "Done — results in $OUTDIR"Complete recon script: input domain, timestamped output directory, sequential enumeration, and vulnerability scanning
chmod +x recon.sh && ./recon.sh example.comMake the script executable and run it against your target — outputs go to a timestamped folder
for domain in $(cat targets.txt); do ./recon.sh $domain; doneBatch mode: run the same recon script against every domain in your target list automatically
#!/bin/bash
TARGET=$1
OUTDIR="recon-$TARGET"
[ -d "$OUTDIR" ] && echo "Error: $OUTDIR exists" && exit 1
mkdir $OUTDIR
subfinder -d $TARGET -silent -o $OUTDIR/subs.txt
echo "Phase 1/4 complete: $(wc -l < $OUTDIR/subs.txt) subdomains"Script with error handling: check for duplicate runs, show progress after each phase with counter
./recon.sh example.com 2>&1 | tee recon.logRun the script and log everything to a file for debugging and audit trail
#!/bin/bash
run_phase() {
local phase=$1
local cmd=$2
echo "[$(date +%H:%M:%S)] Phase $phase started"
eval "$cmd"
echo "[$(date +%H:%M:%S)] Phase $phase finished"
}
run_phase "1-Subs" "subfinder -d $1 -silent -o subs.txt"
run_phase "2-Live" "httpx -l subs.txt -silent -o live.txt"Modular script with function-based phases — timestamped logging for each phase with start/finish markers
TIPS
A production-grade pipeline runs stages in parallel, handles failures gracefully, and notifies you when results are ready. This section builds a complete multi-stage pipeline that handles subdomain enumeration, port scanning, technology detection, vulnerability scanning, and reporting.
#!/bin/bash
TARGET=$1; O="recon-$TARGET-$(date +%Y%m%d)"
mkdir -p $O/{subs,ports,live,vulns,report}
subfinder -d $TARGET -silent -o $O/subs/all.txt
puredns resolve $O/subs/all.txt -r resolvers.txt -o $O/subs/resolved.txt
httpx -l $O/subs/resolved.txt -silent -title -tech-detect -status-code -o $O/live/enriched.txt
naabu -list $(awk '{print $NF}' $O/subs/resolved.txt | sort -u) -top-ports 1000 -silent -o $O/ports/open.txtStage 1: parallel-ready pipeline design — each phase uses a dedicated subdirectory for clean output organization
subfinder -d example.com -silent | dnsx -silent -a -resp-only | naabu -list - -top-ports 100 -silent > ips-ports.txtParallel execution pipe: subdomains → IP resolution → port scan in one memory-efficient pipeline
nuclei -l $O/live/enriched.txt -t cves/ -severity critical,high -silent -o $O/vulns/critical.txtStage 2: vulnerability scanning — focus on critical and high severity CVEs first for maximum impact
nuclei -l $O/live/enriched.txt -t exposures/ -silent -o $O/vulns/exposures.txtStage 3: exposure scanning — check for open S3 buckets, debug pages, admin panels, and misconfigurations
nuclei -l $O/live/enriched.txt -t misconfiguration/ -silent -o $O/vulns/misconfig.txtStage 4: misconfiguration scanning — find security header gaps, directory listing, and other common flaws
cat $O/vulns/*.txt | grep -v '^$' | sort -u > $O/report/all-findings.txtReport generation: merge all vulnerability files into a single deduplicated findings report
wc -l $O/report/all-findings.txt && echo 'vulnerabilities found'Quick summary: count total unique findings and display directly in terminal output
TIPS
A pipeline that runs silently and never tells you anything is useless. Set up notifications so you get alerts when critical vulnerabilities are found. Support multiple channels: a terminal bell for local scripts, Telegram for remote monitoring, and Slack for team collaboration.
echo "Critical vulnerability found!" | notify -silent -provider telegramSend a notification via Telegram using ProjectDiscovery's notify tool — requires pre-configured provider
nuclei -l live.txt -t cves/ -severity critical -json -silent | notify -silentPipe critical Nuclei findings directly to Telegram/Slack — instant alert when something important is found
#!/bin/bash
notify_critical() {
local finding=$1
echo "$finding" | notify -silent -provider telegram
echo "$finding" | notify -silent -provider slack
echo -e "\a" # terminal bell
}
subfinder -d $1 -silent | httpx -silent | nuclei -t cves/ -severity critical -json -silent | while read line; do
notify_critical "$line"
doneMulti-channel alert script: Telegram + Slack + terminal bell when critical CVEs are detected in real-time
notify -provider telegram -silent -data '{"message":"Recon complete for example.com — 23 live hosts, 12 open ports, 3 critical findings"}'Send a formatted summary message at the end of a pipeline run — one digest instead of multiple alerts
nuclei -l live.txt -t cves/ -json -silent | jq -r '[.info.severity, .info.name, .host] | @tsv' | while IFS=$'\t' read sev name host; do echo "Alert: $sev - $name on $host" | notify -silent; doneParse Nuclei JSON output with jq and send individual alerts per finding with severity and host info
cat $OUTDIR/vulns/critical.txt | notify -silent -bulk -id critical-findingsBulk notify: send all critical findings as a single message using a custom provider ID
TIPS
Most recon tools are CPU-bound or I/O-bound on a single thread. GNU Parallel and xargs distribute work across all CPU cores, processing multiple targets simultaneously. A pipeline that takes 1 hour on one core finishes in 2 minutes on 30 cores. This is the single biggest performance optimization you can make.
cat domains.txt | parallel -j 32 'subfinder -d {} -silent | httpx -silent -title -status-code >> results/{}.txt'Parallel subdomain enumeration across 32 cores — each domain gets its own subfinder+httpx pipeline
subfinder -d example.com -silent | parallel -j 50 'dnsx -silent -a -resp-only | httpx -silent -title -status-code'Parallel DNS resolution and HTTP probing — 50 subdomains processed simultaneously
cat targets.txt | xargs -P 10 -I {} ./recon-single.sh {}xargs with 10 parallel processes — runs the recon script for 10 targets at once
seq 1 65535 | parallel -j 100 'naabu -host example.com -p {} -silent 2>/dev/null' | sort -u > all-ports.txtParallel full port scan with naabu — 100 ports at a time, completes a full 65535 scan in seconds
parallel -a urls.txt -j 20 'curl -sI {} | head -1' ::: {} > response-codes.txtParallel HTTP header fetcher — check response codes for 20 URLs simultaneously
#!/bin/bash
scan_domain() {
local d=$1
subfinder -d $d -silent | httpx -silent | nuclei -silent -o "nuclei-$d.txt"
}
export -f scan_domain
cat top-100-targets.txt | parallel -j 10 scan_domain {}Export a bash function and run it with GNU Parallel — full pipeline across 100 targets, 10 at a time
parallel -j 0 --progress 'echo Scanning {}; nmap -sV -p 80,443 {}' :::: ip-list.txtParallel nmap with progress bar — shows real-time completion stats for each target being scanned
TIPS
Store every scan result in a Git repository. Each run creates a commit with timestamped results — you can see exactly what changed, when, and roll back if needed. GitOps turns your recon into a searchable, auditable, collaborative database. Push to GitHub/GitLab for remote backup and team access.
#!/bin/bash
# git-recon.sh — automated recon with Git tracking
TARGET=$1; REPO="$HOME/recon-results/$TARGET"
mkdir -p $REPO; cd $REPO
[ ! -d ".git" ] && git init && git commit --allow-empty -m "init"
subfinder -d $TARGET -silent -o subs.txt
httpx -l subs.txt -silent -title -status-code -tech-detect -o live.txt
nuclei -l live.txt -silent -o nuclei.txt
git add .
git commit -m "recon $(date +%Y-%m-%d_%H:%M)"Auto-commit script: run recon and immediately commit results to local Git repository with timestamp
cd ~/recon-results/example.com && git log --oneline -10View the last 10 recon runs — each commit represents a full scan cycle with timestamp
cd ~/recon-results/example.com && git diff HEAD~1 -- subs.txtCompare subdomains between today's scan and yesterday's — exactly what changed in the diff
cd ~/recon-results/example.com && git diff HEAD~7 -- nuclei.txt | grep '^+' | grep -v '^+++'See all new vulnerabilities found in the last week — grep only the additions in the diff
cd ~/recon-results && git remote add origin git@github.com:user/recon-results.git && git push -u origin mainPush all recon results to a private GitHub repository — remote backup and team collaboration
#!/bin/bash
# push-recon.sh — auto push after every scan
cd ~/recon-results/example.com
git add .
git commit -m "auto-scan $(date +%Y-%m-%d_%H:%M)"
git push origin main 2>&1 || echo "Push failed — will retry next cycle"Auto-push script: commit and push to remote after every scan cycle with automatic retry logic
cd ~/recon-results && grep -r 'admin' --include='live.txt' | grep '200'Search across ALL historical scan data: find every admin panel that ever returned 200 across all targets
TIPS
Run your pipeline without managing a server. AWS Lambda, Google Cloud Run, and GitHub Actions can execute recon tasks on-demand. Provision a cloud function for each tool, chain them via HTTP/webhooks, and only pay for the compute time you use. Serverless approaches excel at burstable workloads like mass scanning.
cat << EOF > .github/workflows/nightly-recon.yml
name: Nightly Recon
on:
schedule:
- cron: "0 2 * * *"
workflow_dispatch:
jobs:
recon:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Install tools
run: |
go install github.com/projectdiscovery/subfinder/v2/cmd/subfinder@latest
go install github.com/projectdiscovery/httpx/cmd/httpx@latest
go install github.com/projectdiscovery/nuclei/v3/cmd/nuclei@latest
- name: Run recon
run: |
subfinder -d example.com -silent | httpx -silent | nuclei -silent -o results.txt
cat results.txt
- name: Commit results
run: |
git config user.name "recon-bot"
git add .
git commit -m "nightly recon $(date +%Y%m%d)" || echo "no changes"
git push
EOFGitHub Actions workflow: scheduled nightly recon on GitHub's infrastructure — free, no server needed
gcloud functions deploy subfinder-scan --runtime go121 --trigger-http --allow-unauthenticated --entry-point ScanDeploy a Google Cloud Function that runs subfinder — serverless HTTP-triggered recon
curl -X POST https://YOUR_CLOUD_FUNCTION_URL -d '{"domain": "example.com"}'Trigger a serverless scan via HTTP POST — integrate into any pipeline with a simple curl call
cat << EOF > Dockerfile.cloudrun
FROM ubuntu:22.04
RUN apt update && apt install -y golang git
RUN go install github.com/projectdiscovery/httpx/cmd/httpx@latest
ENTRYPOINT ["httpx"]
EOFCloud Run container: single-purpose container with httpx — deploy as a stateless microservice
gcloud run deploy httpx-scan --image gcr.io/your-project/httpx-scan --cpu 4 --memory 8Gi --timeout 900Deploy to Cloud Run with 4 CPUs, 8GB RAM, and 15-minute timeout — enough for large target lists
aws lambda create-function --function-name recon-subfinder --runtime go1.x --role arn:aws:iam::xxx:role/lambda-exec --zip-file fileb://function.zipAWS Lambda deployment for subfinder — serverless recon in the AWS ecosystem with IAM role security
TIPS
TOOLS IN THIS CHAPTER
ProjectDiscovery notification tool — sends pipeline results to Telegram, Slack, Discord, and 20+ services
go install -v github.com/projectdiscovery/notify/cmd/notify@latestThe GNU Bourne-Again SHell — the foundation of every automation pipeline on Linux
Pre-installed on most Linux systems. Windows: WSL or Git BashLightweight JSON processor — essential for parsing tool outputs in pipelines
apt install jqGNU Parallel — distribute jobs across all CPU cores for massive speedup
apt install parallelVersion control system — track every scan result with full history and change detection
apt install gitGitHub CLI — automate repository creation, pushing, and management from the terminal
Can be downloaded from https://cli.github.com/You now understand the core automation patterns: Unix pipes, bash scripting, multi-stage pipelines, parallel execution, GitOps version control, serverless cloud deployment, and real-time notifications. These building blocks let you automate any recon workflow. Chapter 2 takes this further with Nuclei's advanced capabilities.
Custom templates, workflows, and fuzzing at scale
Nuclei is the most powerful tool in your automation arsenal. Beyond basic scanning, it supports custom templates, multi-step workflows, HTTP fuzzing, code execution, and conditional logic. This chapter teaches you to write your own templates, build complex workflows that chain multiple checks together, and use Nuclei for targeted fuzzing and brute-forcing.
Nuclei templates are YAML files that define how to detect vulnerabilities. Every template has a request section (what to send) and a matcher section (what to look for in the response). Learn to read templates, understand their structure, and write your own targeted checks.
nuclei -u https://example.com -t cves/ -stats -o scan-results.txtRun all CVE templates with live statistics — shows progress, found, and rate metrics in real-time
nuclei -u https://example.com -t ~/nuclei-templates/ -severity critical,high -o critical-only.txtRun only critical and high severity templates — focus on the most impactful findings first
nuclei -u https://example.com -t ~/nuclei-templates/ -s 200,403,500 -o status-filtered.txtFilter results by HTTP status code — useful when looking for specific response patterns
nuclei -u https://example.com -t ~/nuclei-templates/ -json -o scan.jsonJSON output format — machine-readable output for pipeline processing with jq and other tools
nuclei -u https://example.com -t ~/nuclei-templates/ -rl 150 -rate-limit-minute 60 -o rate-limited.txtRate-limited scan — 150 requests per second, 60 per minute to avoid WAF blocks
nuclei -l live-urls.txt -t ~/nuclei-templates/ -bulk-size 25 -c 10 -o batch-scan.txtBatch scanning across multiple targets — 25 URLs per batch, 10 concurrent hosts
cat nuclei-results.json | jq -r '[.info.severity, .info.name, .host] | @tsv' | column -t -s $'\t'Format JSON results into a clean severity | name | host table for easy triage and reporting
TIPS
Custom Nuclei templates let you check for vulnerabilities specific to your target. Write templates for internal applications, custom software, or unique misconfigurations. A Nuclei template defines one or more HTTP requests and matchers that extract specific patterns from responses.
cat ~/nuclei-templates/cves/2023/CVE-2023-XXXX.yamlRead an existing CVE template to understand the YAML structure: id, info, requests, and matchers sections
cat << EOF > custom-check.yaml
id: custom-config-exposure
info:
name: Custom Config Exposure
severity: medium
description: Checks for exposed configuration file
requests:
- method: GET
path:
- "{{BaseURL}}/config.json"
matchers:
- type: word
words:
- "api_key"
- "database"
- "password"
EOFCustom template from scratch: checks for config.json files containing sensitive keywords like api_key or password
nuclei -u https://target.com -t custom-check.yaml -o custom-results.txtRun your custom template against a target to test it works before adding it to your template library
cat << EOF > multi-endpoint.yaml
id: multi-endpoint-check
info:
name: Multiple Endpoint Check
severity: info
requests:
- method: GET
path:
- "{{BaseURL}}/robots.txt"
- "{{BaseURL}}/sitemap.xml"
- "{{BaseURL}}/.well-known/security.txt"
matchers:
- type: word
words:
- "Disallow"
- "sitemap"
- "security"
EOFMulti-endpoint template: checks multiple paths in a single request block for common security-related files
cat << EOF > conditional-template.yaml
id: conditional-dir-check
info:
name: Directory exists check
severity: info
requests:
- method: GET
path:
- "{{BaseURL}}/{{path}}"
payloads:
path: admin-paths.txt
stop-at-first-match: true
matchers:
- type: status
status:
- 200
- 403
- 401
EOFParameterized template with payload file — iterates over admin-paths.txt and stops on first match (200/403/401)
nuclei -u https://target.com -t my-templates/ -workflows -o workflow-results.txtRun a custom template directory with workflow execution — templates in subdirectories run in sequence
TIPS
Workflows chain multiple templates together with conditional logic. Template A runs first — if it matches, Template B runs. If Template B matches, Template C runs. This enables multi-stage exploitation checks where each step builds on the previous one.
cat << EOF > chain-workflow.yaml
id: xss-to-account-takeover
info:
name: XSS to ATO chaining
author: automation-guide
steps:
- template: xss-detection.yaml
matchers:
- name: xss-found
- template: cookie-grabber.yaml
matchers:
- name: cookie-captured
- template: session-hijack.yaml
conditions:
- xss-found
- cookie-captured
EOFConditional workflow: runs cookie-grabber only if XSS is found, then session-hijack only if cookie is captured
nuclei -u https://target.com -w chain-workflow.yaml -o workflow-results.txtExecute a workflow file — Nuclei processes templates in order with the defined conditions
cat << EOF > tech-then-exploit.yaml
id: tech-to-exploit
info:
name: Tech detection then exploit
steps:
- template: tech-detect/wordpress.yaml
matchers:
- name: wp-version
- template: cves/wordpress/
conditions:
- wp-version
EOFSmart workflow: detect the technology first, then run ALL relevant CVE templates only if the technology is found
cat << EOF > multi-condition.yaml
id: port-and-service
info:
name: Port discovery with service check
steps:
- template: port-scan.yaml
matchers:
- name: port-443
- template: ssl-check.yaml
matchers:
- name: weak-ciphers
- template: heartbleed.yaml
conditions:
- port-443
- weak-ciphers
EOFMulti-condition workflow: runs Heartbleed check only if port 443 is open AND weak ciphers are detected
nuclei -l targets.txt -w workflows/ -o all-workflow-results.txtRun an entire workflow directory against a target list — processes all .yaml files as individual workflows
TIPS
Beyond vulnerability scanning, Nuclei excels at fuzzing and brute-forcing. Use raw HTTP requests with payload substitution to test endpoints, parameters, and authentication. Nuclei fuzzing is faster than ffuf for some use cases because it avoids the overhead of separate HTTP connections.
cat << EOF > dir-fuzz.yaml
id: directory-fuzzing
info:
name: Nuclei directory fuzzing
requests:
- raw:
- |
GET {{BaseURL}}/{{path}} HTTP/1.1
Host: {{Hostname}}
payloads:
path: directory-wordlist.txt
stop-at-first-match: true
matchers:
- type: status
status:
- 200
- 403
- 401
EOFNuclei-based directory fuzzing — reads paths from a wordlist and checks for 200/403/401 responses
nuclei -u https://target.com -t dir-fuzz.yaml -o fuzzed-dirs.txt -rl 100Run the fuzzing template with rate limiting (100 req/s) to avoid triggering WAFs
cat << EOF > param-fuzz.yaml
id: parameter-fuzzing
info:
name: Nuclei parameter fuzzing
requests:
- raw:
- |
GET {{BaseURL}}/api?{{param}}=test HTTP/1.1
Host: {{Hostname}}
payloads:
param: params.txt
matchers:
- type: word
words:
- "error"
- "warning"
- "exception"
- "stack trace"
negative: true
EOFParameter fuzzing template — injects parameter names from a wordlist and checks for error/exception responses
cat ~/nuclei-templates/fuzzing/oauth-brute.yaml | head -30Read an existing brute-force template to understand the pattern — common for OAuth token and session testing
TIPS
Writing Nuclei templates is a skill that compounds over time. Set up a proper development workflow: a directory structure for your custom templates, a testing pipeline that validates them against known targets, and a peer review process. Well-written templates are reusable across hundreds of engagements.
mkdir -p ~/custom-templates/{cves,exposures,tech-detect,misconfig,workflows} && ls ~/custom-templates/Organized template directory structure — categories mirror the official template repo for consistency
cat << EOF > ~/custom-templates/tech-detect/custom-cms.yaml
id: custom-cms-detection
info:
name: Custom CMS Detection
author: hunter
severity: info
description: Detects Custom CMS v3.x by its unique path
requests:
- method: GET
path:
- "{{BaseURL}}/custom-cms/version.php"
matchers:
- type: word
words:
- "Custom CMS v3."
EOFCreate your first custom technology detection template — detect a specific CMS version by its unique path
nuclei -u https://test-target.com -t ~/custom-templates/ -vTest all your custom templates against a known target in verbose mode — see exactly what matched and why
cat << EOF > ~/custom-templates/workflows/cms-chain.yaml
id: cms-chain
info:
name: CMS Detection to Exploit Chain
steps:
- template: ~/custom-templates/tech-detect/custom-cms.yaml
matchers:
- name: cms-detected
- template: ~/nuclei-templates/cves/2023/*cms*
conditions:
- cms-detected
EOFChain workflow: detect the CMS first → automatically run all matching CVE templates if found
nuclei -u https://target.com -t ~/custom-templates/ -validateValidate ALL your templates for YAML syntax errors before using them in a real engagement
nuclei -u https://target.com -t ~/custom-templates/ -duc -o only-new-findings.txtDeduplicate: skip findings from previous runs (-duc) so you only see NEW vulnerabilities
cat << EOF >> ~/.bash_aliases
alias ncv="nuclei -validate"
alias ncs="nuclei -duc -silent"
EOF && source ~/.bash_aliasesShell aliases for common Nuclei operations — validate templates with ncv and scan with -duc via ncs
TIPS
Embed recon directly into your CI/CD pipeline. Every time you push code, your infrastructure gets scanned automatically. Pull requests trigger targeted scans of new endpoints. This catches vulnerabilities within minutes of deployment — before they reach production and before attackers find them.
cat << EOF > .github/workflows/recon-on-push.yml
name: PR Recon Scan
on:
pull_request:
branches: [main]
jobs:
recon:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Install tools
run: |
go install github.com/projectdiscovery/httpx/cmd/httpx@latest
go install github.com/projectdiscovery/nuclei/v3/cmd/nuclei@latest
- name: Scan new endpoints
run: |
git diff --name-only origin/main | grep -E "html|js|php" > changed-files.txt
cat changed-files.txt | httpx -silent | nuclei -silent -o pr-findings.txt || true
cat pr-findings.txt
EOFPR-triggered recon: scans only the files changed in a pull request for vulnerabilities — targeted and fast
cat << EOF > .github/workflows/weekly-full-scan.yml
name: Weekly Full Infrastructure Scan
on:
schedule:
- cron: "0 6 * * 1"
jobs:
scan:
runs-on: ubuntu-latest
steps:
- name: Full recon scan
run: |
subfinder -d example.com -silent > subs.txt
httpx -l subs.txt -silent -o live.txt
nuclei -l live.txt -silent -o vulns.txt
- name: Upload results
uses: actions/upload-artifact@v4
with:
name: scan-results
path: "*.txt"
- name: Notify on findings
if: failure()
run: echo "Scan failed — check logs" | notify -silent
EOFWeekly full scan via GitHub Actions with artifact storage and failure notification
gh workflow run nightly-recon.yml -f domain=example.comTrigger a workflow manually via GitHub CLI — useful for ad-hoc scans without opening the browser
gh run list --workflow=nightly-recon.yml --limit 5 --json conclusion,createdAt,displayTitleView the last 5 workflow runs with their status and timestamps — monitor scan health from terminal
cat << EOF > .github/workflows/auto-triage.yml
name: Auto-Triage Findings
on:
workflow_run:
workflows: ["Nightly Recon"]
types:
- completed
jobs:
triage:
runs-on: ubuntu-latest
if: ${{ github.event.workflow_run.conclusion == "success" }}
steps:
- name: Download artifacts
uses: actions/download-artifact@v4
- name: Analyze findings
run: |
cat vulns.txt | grep -i "critical" > critical.txt
if [ -s critical.txt ]; then
echo "CRITICAL FINDINGS DETECTED" | notify -silent
cat critical.txt | notify -silent
fi
EOFAuto-triage pipeline: after recon completes, auto-analyze findings and alert only on critical severity
TIPS
Over time, build your own private template library targeting specific platforms, software, and misconfigurations you encounter frequently. A curated collection of 50 well-written templates is worth more than 10,000 generic ones. Prioritize templates for bug bounty programs you hunt on regularly.
nuclei -update-templates && nuclei -silent -t ~/nuclei-templates/ -stats -o full-scan.txtUpdate the official template repo and run a full baseline scan — capture everything as your starting point
cat full-scan.txt | awk '{print $2}' | sort | uniq -c | sort -rn | head -20Analyze which templates trigger most often — identify the patterns that actually matter for your targets
mv ~/nuclei-templates/cves/2020/ ~/nuclei-templates-archive/Prune outdated templates (4+ year old CVEs) — they add noise and rarely apply to modern targets
#!/bin/bash
# organize-templates.sh — keep only useful templates
cd ~/nuclei-templates
mkdir -p active/
cp cves/*/*[Ww]ordpress* active/
cp cves/*/*[Ll]aravel* active/
cp cves/*/*[Aa]pache* active/
cp exposures/* active/
cp misconfiguration/* active/
echo "Archived $(find . -name "*.yaml" | wc -l) templates to active/"Template curation script: extract only templates relevant to your targets (WordPress, Laravel, Apache, etc.)
cat << EOF > ~/custom-templates/misconfig/debug-endpoints.yaml
id: debug-endpoint-check
info:
name: Debug Endpoint Exposure
severity: medium
description: Check for exposed debug endpoints
requests:
- method: GET
path:
- "{{BaseURL}}/debug"
- "{{BaseURL}}/phpinfo.php"
- "{{BaseURL}}/.env"
- "{{BaseURL}}/info.php"
matchers:
- type: word
words:
- "PHP Version"
- "DB_HOST"
- "APP_ENV"
- "xdebug"
EOFCustom debug endpoint scanner — checks multiple debug paths and matches on known informational responses
nuclei -l live.txt -t ~/custom-templates/ -author hunter -o my-findings.txtRun only templates created by you (filter by author) — useful when testing newly written templates
TIPS
TOOLS IN THIS CHAPTER
Fast vulnerability scanner with 10,000+ templates, custom templates, workflows, and fuzzing
go install -v github.com/projectdiscovery/nuclei/v3/cmd/nuclei@latestOfficial template repository — 10,000+ templates for CVEs, exposures, misconfigurations, and tech detection
git clone https://github.com/projectdiscovery/nuclei-templates.git ~/nuclei-templatesYAML syntax highlighting and validation makes writing custom templates much easier
Can be downloaded from https://code.visualstudio.com/downloadYou now understand Nuclei at an expert level: reading templates, writing your own, building conditional workflows, and fuzzing endpoints. Chapter 3 takes automation further with cron jobs, Docker, and a fully automated monitoring infrastructure.
Run your pipelines automatically on a schedule and never miss a change
The most powerful recon setup runs continuously. Instead of scanning once, you scan every day and compare results. New subdomains, open ports, and technologies are detected automatically as they appear. This chapter teaches you to deploy your pipeline on a VPS, schedule it with cron and systemd, run it in Docker containers, and monitor results over time with change detection.
Cron is the simplest and most reliable scheduler on Linux. Set your pipeline to run daily, weekly, or hourly. Combine multiple cron jobs for different pipeline stages — subdomain enum every 6 hours, port scan daily, vulnerability scan weekly.
crontab -eEdit your cron jobs — opens the crontab file in your default editor (nano/vim)
0 */6 * * * /home/user/scripts/recon.sh example.com >> /var/log/recon.log 2>&1Run recon pipeline every 6 hours — logs all output to a central file for debugging
0 2 * * * /home/user/scripts/nuclei-scan.sh example.comRun vulnerability scan daily at 2 AM — off-peak hours for minimal target disruption
0 0 * * 0 /home/user/scripts/full-recon.sh example.comFull recon scan every Sunday at midnight — comprehensive weekly review of the target's attack surface
*/30 * * * * /home/user/scripts/check-new-subs.sh example.comCheck for new subdomains every 30 minutes — rapid detection of newly deployed assets
0 6 * * 1 /home/user/scripts/diff-report.sh example.comGenerate a diff report every Monday at 6 AM — compares this week's results with last week's
cat /var/log/recon.log | grep -E 'critical|high|ERROR' | tail -20Quick check: view the most recent critical findings and errors from your cron logs
#!/bin/bash
# check-cron.sh — verify all cron jobs are running
echo "Cron jobs for $(whoami):"
crontab -l
# latest log entries
tail -5 /var/log/recon.logHealth check script: verify cron jobs are registered and the latest logs show recent activity
TIPS
Systemd timers are more reliable than cron for complex pipelines. They support dependencies (Timer B starts only after Service A finishes), randomized delays, and persistent state across reboots. Use systemd for production-grade scheduled scanning.
cat << EOF | sudo tee /etc/systemd/system/recon-pipeline.service
[Unit]
Description=Recon Pipeline Service
After=network.target
[Service]
Type=oneshot
ExecStart=/home/user/scripts/recon.sh example.com
User=user
Group=user
EOFCreate a systemd service unit — defines what command to run as a systemd-managed service
cat << EOF | sudo tee /etc/systemd/system/recon-pipeline.timer
[Unit]
Description=Run recon every 6 hours
[Timer]
OnCalendar=*-*-* 0,6,12,18:00:00
RandomizedDelaySec=300
Persistent=true
[Install]
WantedBy=timers.target
EOFCreate a systemd timer unit — schedules the service to run at 0/6/12/18 with randomized delay for stealth
sudo systemctl daemon-reload && sudo systemctl enable recon-pipeline.timer && sudo systemctl start recon-pipeline.timerReload systemd, enable the timer to start on boot, and activate it immediately
sudo systemctl status recon-pipeline.timer && sudo systemctl list-timers --all | grep reconVerify the timer is active — shows next run time, last run time, and status
sudo journalctl -u recon-pipeline.service -fFollow the service logs in real-time — useful for debugging pipeline execution
sudo journalctl -u recon-pipeline.service --since '1 hour ago' | grep -E 'error|critical|finding'Filter service logs for the past hour — grep for important keywords in pipeline output
cat << EOF > /etc/systemd/system/nuclei-scan.service
[Unit]
Description=Weekly nuclei vulnerability scan
After=recon-pipeline.service
[Service]
Type=oneshot
ExecStart=/home/user/scripts/nuclei-scan.sh
EOFDependency-based service: this scan runs AFTER the recon pipeline completes successfully
TIPS
Docker containers provide a reproducible environment for your pipelines. No more dependency issues, Go version conflicts, or missing tools. Build a Docker image with all your tools pre-installed and run it anywhere — locally, on a VPS, or in CI/CD.
cat << EOF > Dockerfile
FROM ubuntu:22.04
RUN apt update && apt install -y golang git curl jq ca-certificates
RUN go install -v github.com/projectdiscovery/subfinder/v2/cmd/subfinder@latest
RUN go install -v github.com/projectdiscovery/httpx/cmd/httpx@latest
RUN go install -v github.com/projectdiscovery/nuclei/v3/cmd/nuclei@latest
RUN go install -v github.com/projectdiscovery/notify/cmd/notify@latest
ENV PATH="/root/go/bin:${PATH}"
WORKDIR /recon
COPY scripts/ .
ENTRYPOINT ["./recon.sh"]
EOFDockerfile with all recon tools pre-installed — single image that contains your entire pipeline toolchain
docker build -t recon-pipeline .Build the Docker image — downloads dependencies and installs all tools in a reproducible environment
docker run --rm -v $(pwd)/output:/recon/output recon-pipeline example.comRun the pipeline container — mounts a local output directory so results persist after the container exits
cat << EOF > docker-compose.yml
version: "3.8"
services:
recon:
build: .
volumes:
- ./output:/recon/output
- ./config:/recon/config
command: ["./recon.sh", "example.com"]
EOFDocker Compose configuration — simplifies volume mounts and command arguments
docker-compose up && docker-compose downRun via Docker Compose and clean up — containers are ephemeral, only results persist
cat << EOF > .dockerignore
node_modules
.git
*.md
output/temp
EOFDocker ignore file — prevents unnecessary files from being copied into the image, keeping it small
TIPS
Scanning once gives you a snapshot. Scanning every day gives you a timeline. Use diff tools to compare today's results with yesterday's — new subdomains, new ports, new technologies, and new vulnerabilities appear as changes. This is how you find zero-day assets that no one else has discovered yet.
#!/bin/bash
# diff-recon.sh — compare recon runs
TODAY=reports/$(date +%Y%m%d)
YESTERDAY=reports/$(date -d "yesterday" +%Y%m%d)
if [ -d "$YESTERDAY" ]; then
echo "=== New Subdomains ==="
diff $YESTERDAY/subs.txt $TODAY/subs.txt | grep "^>" | wc -l
echo "=== Removed Subdomains ==="
diff $YESTERDAY/subs.txt $TODAY/subs.txt | grep "^<" | wc -l
fiChange detection script: compare today's subdomains with yesterday's — find what changed
diff --unchanged-line-format= --old-line-format="[-%L]" --new-line-format="[+%L]" old-subs.txt new-subs.txt > subs-changes.txtFormatted diff output — marks removed entries with [-] and new entries with [+] for clear reading
cat new-subs.txt | while read sub; do grep -q $sub old-subs.txt || echo $sub >> unique-new-subs.txt; doneSimple bash loop: find subdomains in today's scan that weren't in yesterday's scan
comm -13 old-sorted.txt new-sorted.txt > truly-new.txtFast set comparison with comm — line 3 shows entries only in new-sorted.txt (requires sorted input)
cat new-subs.txt | grep -vxFf old-subs.txt > brand-new-subs.txtgrep-based set subtraction: find lines in new that don't exist in old — simple and effective
notify -silent -data '{"message":"New subdomain detected: admin-staging.example.com"}' -provider telegramAlert on change detection: send an instant Telegram notification when a new high-value subdomain is found
TIPS
Your automated pipeline needs a home. A $5/month VPS is enough to run continuous recon for multiple targets. This section covers VPS selection, initial setup, security hardening, and monitoring — everything you need to deploy a production-grade recon infrastructure.
ssh root@YOUR_VPS_IP && apt update && apt upgrade -yInitial VPS setup: SSH into the server and update all packages to the latest versions
adduser hunter && usermod -aG sudo hunterCreate a dedicated user for recon operations — never run pipelines as root for security
ssh-keygen -t ed25519 -f ~/.ssh/recon-key && ssh-copy-id -i ~/.ssh/recon-key hunter@YOUR_VPS_IPGenerate an Ed25519 SSH key and copy it to the VPS for passwordless authentication
echo "Port 2222
PermitRootLogin no
PasswordAuthentication no
PubkeyAuthentication yes" | sudo tee /etc/ssh/sshd_config.d/hardening.confSSH hardening: custom port (2222), disable root login and passwords, require key-based auth
sudo ufw allow 2222/tcp && sudo ufw allow 80,443/tcp && sudo ufw enableFirewall: allow SSH on custom port, allow outbound HTTP/HTTPS, block everything else
sudo apt install -y docker.io docker-compose && sudo usermod -aG docker hunterInstall Docker and add your user to the docker group so you can run containers without sudo
sudo apt install -y fail2ban && sudo systemctl enable fail2banInstall fail2ban to automatically block IPs with repeated failed SSH login attempts
go install -v github.com/projectdiscovery/subfinder/v2/cmd/subfinder@latest && go install -v github.com/projectdiscovery/httpx/cmd/httpx@latest && go install -v github.com/projectdiscovery/nuclei/v3/cmd/nuclei@latest && go install -v github.com/projectdiscovery/notify/cmd/notify@latestInstall all core recon tools on the VPS — run once during initial setup
TIPS
Raw scan output is useful, but visual reports communicate findings much more effectively — especially when sharing with team members or including in bug bounty submissions. Build automated HTML reports that summarize every scan run, highlight new findings, and provide drill-down access to raw data.
#!/bin/bash
# generate-report.sh — build an HTML report from scan output
cat << EOF > report.html
<html><head><title>Recon Report - $(date +%Y-%m-%d)</title>
<style>
body { font-family: monospace; background: #1a1a2e; color: #e0e0e0; padding: 2rem; }
h1 { color: #ff6b6b; border-bottom: 2px solid #ffd93d; }
h2 { color: #ffd93d; }
.vuln-critical { color: #ff4757; font-weight: bold; }
.vuln-high { color: #ff6b81; }
.vuln-medium { color: #ffa502; }
.vuln-low { color: #2ed573; }
.section { background: #16213e; padding: 1rem; border-radius: 8px; margin: 1rem 0; }
.summary { display: flex; gap: 2rem; }
.stat { text-align: center; background: #0f3460; padding: 1rem; border-radius: 8px; min-width: 120px; }
.stat-value { font-size: 2rem; font-weight: bold; color: #ffd93d; }
EOF
echo "<div class=\"summary\">" >> report.html
for cat in critical high medium low; do
count=$(grep -ci "$cat" scan-results.txt 2>/dev/null || echo 0)
echo "<div class=\"stat\"><div class=\"stat-value\">$count</div><div>$cat</div></div>" >> report.html
done
echo "</div></body></html>" >> report.html
scp report.html user@vps:/var/www/html/recon-report.html
echo "Report generated: report.html"Generate a self-contained HTML report with severity distribution stats — scp it to a web server for live viewing
cat scan-results.txt | sort -t'[' -k2 | awk -F'[][]' '{print $2, $0}' | sort -rn > sorted-by-severity.txtSort findings by severity — extract the severity label and sort descending so critical issues appear first
cat scan-results.txt | awk -F'[][]' '{a[$2]++} END {for(s in a) print s, a[s]}' | sort -k2 -rnSeverity distribution summary — count how many critical, high, medium, and low findings per scan run
nuclei -l live.txt -o json-output.json -json -silentOutput Nuclei results in JSON format — machine-readable for programmatic processing and dashboarding
cat json-output.json | jq -r '[.info.severity, .info.name, .host] | @tsv' | column -t -s $'\t'Parse JSON results into a clean aligned table with jq — severity, template name, and affected host
cat << EOF > ~/scripts/trend-report.sh
#!/bin/bash
# Track weekly finding counts across all targets
TARGETS=(
"example.com"
"test.com"
"demo.org"
)
for domain in "${TARGETS[@]}"; do
echo "=== $domain ==="
total=$(find reports/$domain -name "vulns.txt" -exec cat {} + | grep -v "^$" | wc -l)
echo "Total findings (all time): $total"
this_week=$(find reports/$domain -name "vulns.txt" -mtime -7 -exec cat {} + | grep -v "^$" | wc -l)
echo "This week: $this_week"
done
EOF && chmod +x ~/scripts/trend-report.shTrend analysis script: report total and weekly finding counts across all bug bounty targets
TIPS
Your VPS has limited disk space. For long-term data retention, offload scan results to object storage. DigitalOcean Spaces, AWS S3, or Backblaze B2 cost pennies per month and provide unlimited storage for historical scan data. This section covers automated uploads and data archival.
sudo apt install s3cmd && s3cmd --configureInstall s3cmd (works with S3, DO Spaces, and MinIO) and run the interactive configuration wizard
s3cmd mb s3://recon-results-$(date +%Y%m) --region=nyc3Create a monthly bucket for recon results — each month gets its own bucket for organization
s3cmd sync reports/ s3://recon-results/$(date +%Y%m)/ --delete-removedSync local reports directory to cloud storage — uploads new files and removes deleted ones
s3cmd put scan-results.tar.gz s3://recon-archive/$(date +%Y%m%d)-scan.tar.gzUpload a compressed archive of scan results — tar.gz reduces storage by 80-90%
s3cmd ls s3://recon-results/ --recursive | tail -20List the most recent files in cloud storage — verify uploads are working correctly
#!/bin/bash
# daily-archive.sh — compress and upload
TIMESTAMP=$(date +%Y%m%d_%H%M)
tar -czf /tmp/scan-$TIMESTAMP.tar.gz reports/$TIMESTAMP/
s3cmd put /tmp/scan-$TIMESTAMP.tar.gz s3://recon-archive/
rm /tmp/scan-$TIMESTAMP.tar.gzDaily archive script: compress, upload, and clean up — keeps VPS storage usage under control
s3cmd setacl s3://recon-archive/20250711-scan.tar.gz --acl-publicMake specific results publicly accessible — useful for sharing findings with program teams or collaborators
cat << EOF >> ~/scripts/cron-jobs.sh
# Daily archive to cloud storage at 6 AM
0 6 * * * /home/hunter/scripts/daily-archive.sh
EOFSchedule automatic daily cloud backups — add to your existing cron-jobs.sh and never lose results
TIPS
When you're running recon on 20+ targets, you need a centralized dashboard to monitor scan health, finding trends, and infrastructure status. Grafana connected to a Prometheus metrics exporter gives you real-time visibility into your entire recon pipeline — all from a single web interface.
cat << EOF > docker-compose-monitoring.yml
version: "3.8"
services:
prometheus:
image: prom/prometheus:latest
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
- prometheus-data:/prometheus
ports:
- "9090:9090"
grafana:
image: grafana/grafana:latest
volumes:
- grafana-data:/var/lib/grafana
ports:
- "3000:3000"
depends_on:
- prometheus
volumes:
prometheus-data:
grafana-data:
EOFDocker Compose for Prometheus + Grafana — spin up a full monitoring stack in seconds
cat << EOF > prometheus.yml
scrape_configs:
- job_name: "recon-metrics"
static_configs:
- targets: ["localhost:8000"]
EOFPrometheus scrape config — pull metrics from your recon metrics exporter running on port 8000
cat << EOF > ~/scripts/metrics-exporter.py
#!/usr/bin/env python3
from http.server import HTTPServer, BaseHTTPRequestHandler
import subprocess, os
class MetricsHandler(BaseHTTPRequestHandler):
def do_GET(self):
subs = int(subprocess.getoutput("cat reports/latest/subs.txt 2>/dev/null | wc -l") or 0)
live = int(subprocess.getoutput("cat reports/latest/live.txt 2>/dev/null | wc -l") or 0)
vulns = int(subprocess.getoutput("cat reports/latest/vulns.txt 2>/dev/null | wc -l") or 0)
disk = subprocess.getoutput("df / --output=pcent | tail -1 | tr -d ' %'")
metrics = f"""# HELP recon_subs_total Total subdomains discovered
# TYPE recon_subs_total gauge
recon_subs_total {subs}
# HELP recon_live_total Live hosts found
# TYPE recon_live_total gauge
recon_live_total {live}
# HELP recon_vulns_total Total vulnerabilities found
# TYPE recon_vulns_total gauge
recon_vulns_total {vulns}
# HELP recon_disk_usage VPS disk usage percent
# TYPE recon_disk_usage gauge
recon_disk_usage {disk}
"""
self.send_response(200)
self.send_header("Content-type", "text/plain")
self.end_headers()
self.wfile.write(metrics.encode())
HTTPServer(("0.0.0.0", 8000), MetricsHandler).serve_forever()
EOFPrometheus metrics exporter — exposes subdomain count, live hosts, vulnerabilities, and disk usage as Prometheus metrics
docker compose -f docker-compose-monitoring.yml up -dStart the monitoring stack in detached mode — Prometheus on :9090, Grafana on :3000
cat << EOF > ~/scripts/start-monitoring.sh
#!/bin/bash
# Start the metrics exporter and monitoring stack
python3 ~/scripts/metrics-exporter.py &
sleep 2
docker compose -f ~/docker-compose-monitoring.yml up -d
echo "Grafana: http://localhost:3000 (admin/admin)"
echo "Prometheus: http://localhost:9090"
EOF && chmod +x ~/scripts/start-monitoring.shOne-command monitoring startup — launches the metrics exporter and the full monitoring stack
curl -s http://localhost:8000 | grep -E "^recon_"Verify the metrics exporter is working — should return current subdomain count, live hosts, vuln count, and disk usage
echo "Alerting rule added: if subdomain count drops by 50% in 24h, check for VPS/data corruption"Create a Prometheus alert: sudden drop in subdomain count likely indicates data loss or scan failure
TIPS
TOOLS IN THIS CHAPTER
Job scheduler for Unix-like systems — runs your pipelines on a time-based schedule
System and service manager for Linux — more reliable than cron with dependency support
Container platform for reproducible pipeline environments across any infrastructure
https://docs.docker.com/engine/install/Uncomplicated Firewall — simple firewall management for securing your VPS
sudo apt install ufwIntrusion prevention tool that blocks brute-force SSH attacks automatically
sudo apt install fail2banCommand-line S3 client for cloud storage uploads — works with AWS S3, DO Spaces, MinIO
sudo apt install s3cmdOpen-source analytics and monitoring dashboard — visualize scan metrics and trends
Run via Docker: grafana/grafana:latestMetrics collection and alerting toolkit — stores scan metrics for real-time querying
Run via Docker: prom/prometheus:latestYou now have a fully automated recon infrastructure: cron/systemd scheduling, Docker reproducibility, change detection, cloud storage backups, a hardened VPS, and a Grafana monitoring dashboard. Your pipeline runs continuously, detects new assets automatically, and alerts you in real-time. This is the infrastructure that separates professional bug hunters from beginners.