Tools menu
Free tool
Robots.txt Breakage Detector
Detect potential issues in robots.txt that could block important pages from search engine crawlers.
Googlebot follows the * group · 3 of 6 blocked
| / | Allowed | no matching rule |
| /services/web-design/ | Allowed | no matching rule |
| /admin/settings | Blocked | line 2: Disallow: /admin/ |
| /admin/help/faq | Allowed | line 5: Allow: /admin/help/ |
| /search?q=seo | Blocked | line 3: Disallow: /search |
| /blog/?sort=new | Blocked | line 4: Disallow: /*?sort= |
Problems in this robots.txt
- Line 8: Disallow: / blocks the whole site for gptbot.
Matching follows Google's rules: the most specific user-agent group applies, the longest matching path wins, and Allow wins a tie. Blocking a URL stops crawling, not indexing; use noindex to keep a page out of results.
How to use this tool
Follow these steps to get the most out of the Robots.txt Breakage Detector.
Enter your domain URL.
The tool scans for common logic errors in robots.txt.
Identify conflicting Allow/Disallow rules.
Check for accidental blocking of CSS/JS resources.
Receive a clean bill of health or a fix list.
Want it done for you?
Crawling and indexing problems quietly cap your rankings. Our technical SEO audits find and fix them.
SEO ServicesMore in Sitemaps & Robots