Robots.txt Generator
Build a valid robots.txt with allow/disallow rules, sitemap and crawl delay.
Upload the result as robots.txt in your site root.
About this tool
The robots.txt generator builds a standards-compliant robots.txt file with per user-agent allow/disallow rules, an optional crawl delay and your sitemap location. A correct robots.txt keeps crawlers out of admin areas and duplicate traps while guiding them to your important pages.
How to use
- 1. Fill in user-agents, disallowed and allowed paths.
- 2. Add your sitemap URL and an optional crawl delay.
- 3. Copy the output and upload it as /robots.txt on your domain.
Frequently asked questions
Does robots.txt block all bad bots?
No, it's advisory. Well-behaved crawlers obey it; malicious scrapers ignore it. Never rely on it for security.
Should I disallow anything by default?
Admin, login, cart and internal search pages are common candidates.
Where does robots.txt go?
In your domain root: https://example.com/robots.txt.
Is your website secure? Run the free 7-section scan: vulnerabilities, malware, headers, SEO and speed.
Scan my websiteRelated tools
More from Technical SEO and SEO.