Tools menu
Free tool
Robots.txt Validator
Validate your robots.txt file for syntax errors and check crawl permissions for different user agents.
Crawl Guardian
Robots.txt Power Pack
User-agent: *
Disallow: /admin/
Allow: /
Sitemap: https://example.com/sitemap.xmlBot Governance Standard
One wrong Disallow line can stop Google crawling your whole site. Use these templates as a starting point and test the result before you publish. robots.txt controls crawling, not indexing, and it does not hide private pages: protect those with a login or noindex.
How to use this tool
Follow these steps to get the most out of the Robots.txt Validator.
Paste your current robots.txt content or upload the file.
Enter a test URL to check against your rules.
Select the User-Agent you wish to simulate.
Run the test to see if the URL is Allowed or Blocked.
Fix any unintended blocking rules immediately.
In depth
Bulletproof Crawl Control
Robots.txt is the first file search engine bots look for. A single wrong rule, like Disallow: / under User-agent: *, can block crawlers from your whole site.
Starting Templates
It includes starting templates for WordPress and Shopify. On Shopify, robots.txt is edited through the robots.txt.liquid theme template rather than uploaded as a file.
Want it done for you?
Crawling and indexing problems quietly cap your rankings. Our technical SEO audits find and fix them.
SEO ServicesMore in Sitemaps & Robots