Tools menu
Free tool
Crawl Simulation Checker
Simulate how search engine bots crawl your website based on your robots.txt configuration.
Googlebot follows the * group · 3 of 6 blocked
| / | Allowed | no matching rule |
| /services/web-design/ | Allowed | no matching rule |
| /admin/settings | Blocked | line 2: Disallow: /admin/ |
| /admin/help/faq | Allowed | line 5: Allow: /admin/help/ |
| /search?q=seo | Blocked | line 3: Disallow: /search |
| /blog/?sort=new | Blocked | line 4: Disallow: /*?sort= |
Problems in this robots.txt
- Line 8: Disallow: / blocks the whole site for gptbot.
Matching follows Google's rules: the most specific user-agent group applies, the longest matching path wins, and Allow wins a tie. Blocking a URL stops crawling, not indexing; use noindex to keep a page out of results.
How to use this tool
Follow these steps to get the most out of the Crawl Simulation Checker.
Input your Robots.txt rules and a target URL.
Simulate a visit from Googlebot Smartphone.
Watch the step-by-step logic matching process.
Confirm exactly which line triggers a block/allow.
Verify your strict indexing controls.
Want it done for you?
Crawling and indexing problems quietly cap your rankings. Our technical SEO audits find and fix them.
SEO ServicesMore in Sitemaps & Robots