Robots.txt Generator - aitoolsweb.org

General Settings

Crawl-Delay (Optional)

Time in seconds bots should wait between requests.

Sitemap URL

WordPress Rules

Disallow /wp-admin/ Allow /wp-admin/admin-ajax.php Disallow /wp-includes/ Disallow /wp-content/plugins/ Disallow /wp-content/themes/

Extra Rules

Custom Directories (One per line)

These will be added as 'Disallow'.

Generated Output

Copied to clipboard!