Robots.txt Generator
Build a robots.txt draft from a sitemap URL, allow and disallow paths, and an optional crawl delay.
A QUICK WALKTHROUGH
How to use this tool
- Enter the sitemap URL and paths, then choose an optional delay.
- Review the text and copy it to the root robots.txt for the same site as the sitemap.
Scope and limits
This creates one User-agent: * group. It does not fetch or verify the sitemap, guess a domain, control access, or guarantee crawling or indexing. Enter an absolute http:// or https:// sitemap URL and valid paths beginning with /. Use up to 100 paths, 2,048 URL characters, and 500 characters per path. Crawl delay must be a whole number from 0 to 120.
Standards note
Crawl-delay is a non-standard extension. Google Search does not support it; use it only when the crawler documentation for your target supports it.
GOOD TO KNOW
Common questions
Does this publish or check my site?
No. The URL and rules stay in this browser; the tool only formats your entries.