Uro

URL deduplicator and cleaner that removes duplicates and low-quality paths from URL lists

Recon & OSINT7 commands

#url#deduplicate#cleanup#python#recon

Installation

1Install via pip

2Verify installation

# Using pip
pip install uro

# Verify
uro --help

# Update
pip install --upgrade uro

Basic Usage

Deduplicate and clean URL lists by removing duplicates, similar paths, and low-value entries

# Deduplicate URLs from stdin
cat urls.txt | uro

# Deduplicate and sort by path depth
cat urls.txt | uro --depth

# Only keep unique paths (ignore query params)
cat urls.txt | uro --unique

# Output to file
cat urls.txt | uro > clean-urls.txt

# Include subdomains
cat urls.txt | uro --subs

# Show statistics
cat urls.txt | uro --stats

# Keep URLs with specific extensions
cat urls.txt | uro --include php,asp,jsp

# Exclude specific extensions
cat urls.txt | uro --exclude png,jpg,css

Command Reference

7 commands

1Sort by path depth

--depth

2Keep only unique paths

--unique

3Include subdomain URLs

--subs

4Show deduplication statistics

--stats

5Keep only URLs with these extensions

--include

6Remove URLs with these extensions

--exclude

7Group similar URLs

--similar

When to Use

Cleaning up large URL lists before vulnerability scanning

Removing duplicate entries from multiple recon sources

Reducing noise by excluding static file extensions

Preparing clean input for tools like gf or dalfox

Analyzing URL dataset statistics and patterns

Notes & Tips

Reduces URL list size significantly by removing duplicates

Can filter by file extension to focus on dynamic pages

Use --depth to prioritize shallow paths for manual review

Works well in pipelines with gau, waybackurls, and other URL tools

Common Errors & Solutions

Empty output

Check input format — each URL should be on a separate line

Too aggressive filtering

Remove --exclude or adjust include/exclude lists

# Deduplicate URLs from stdin cat urls.txt | uro # Deduplicate and sort by path depth cat urls.txt | uro --depth # Only keep unique paths (ignore query params) cat urls.txt | uro --unique # Output to file cat urls.txt | uro > clean-urls.txt # Include subdomains cat urls.txt | uro --subs # Show statistics cat urls.txt | uro --stats # Keep URLs with specific extensions cat urls.txt | uro --include php,asp,jsp # Exclude specific extensions cat urls.txt | uro --exclude png,jpg,css