robots.txt Tester
Paste robots.txt text to inspect its directive groups and find common syntax problems locally.
A QUICK WALKTHROUGH
How to use this tool
- robots.txt text
- Inspect locally
Scope and limits
This checker reads only the text you paste. It does not fetch a site or test live crawling. It checks common User-agent, Allow, Disallow, Sitemap and Host syntax, group structure, duplicate directives, and unsupported lines; it is not a complete implementation of every Google or robots exclusion standard.
GOOD TO KNOW
Common questions
robots.txt text
This checker reads only the text you paste. It does not fetch a site or test live crawling. It checks common User-agent, Allow, Disallow, Sitemap and Host syntax, group structure, duplicate directives, and unsupported lines; it is not a complete implementation of every Google or robots exclusion standard.