SEO & Marketing
Robots.txt Tester
Read the live robots.txt, identify general allow and disallow rules, discover declared sitemaps and test them against real public paths.
Preparing the release cockpit…
Share this tool
Referencing this tool on a website?
Copy an attribution link. It is optional and does not change free access.
Transparent methodology
What the result includes—and what it does not
Built for developers checking whether a live public path is affected by general robots.txt rules.
Calculation method
- 1Fetch robots.txt at the target origin and parse User-agent, Allow, Disallow and Sitemap directives.
- 2Apply the longest matching rule from the general asterisk group to the checked path.
- 3Cross blocked paths with public-scope and sitemap intent, while keeping malformed lines visible.
Assumptions and limits
- This mode tests the general user-agent group. Individual search crawlers can select a more specific group and support provider-specific directives.
- robots.txt is a crawl-control mechanism and does not guarantee removal from results.
- The parser does not claim parity with every undocumented crawler behavior.
Worked example
Product directory blocked after staging
Input: A public /products/item URL with Disallow: /products/ in the general group.
Output: Block release with the matched public URL and the instruction to allow the intended path or exclude it deliberately.
Primary sources
- Google Search — Crawling and indexingPrimary documentation for crawl access, indexability signals, canonicals, sitemaps, redirects and JavaScript search behavior.
- Google Search — robots.txtExplains that robots.txt primarily controls crawling and is not a guaranteed removal mechanism.
- Google Search — Canonical URLsOfficial guidance for canonical signals and duplicate URL consolidation.
- Sitemaps protocolOpen protocol for sitemap XML, sitemap indexes and loc elements.
How to use it
From public URL to verified release.
- 1Choose the release mission and declare whether the scope should be public.
- 2Keep the page open while respectful batches read public HTML and cross page signals.
- 3Export the evidence, deploy the fix, then load the project to verify the same scope.
Built for trust
Evidence without a false promise.
- No registration or paywall before your result.
- Only public URLs are fetched; HTML is not written into the project.
- Ready to ship describes the checked scope; it never guarantees indexing or ranking.
Frequently asked questions
How do I test a URL against robots.txt?
Enter the complete public URL. The tool fetches the origin's robots.txt, reads the general user-agent group and applies the longest matching Allow or Disallow pattern to that path.
Does Allow override Disallow?
The most specific matching rule determines the result in the model used here, so the longer matching path wins. A crawler-specific group or implementation can still differ and should be checked in that provider's official tester.
Why can a robots-blocked URL still appear in search?
Because blocking a request does not erase knowledge of the URL. Search engines can discover it through links, and the block can stop them from reading a noindex directive placed inside the page.
Keep solving
Related seo & marketing tools
Website Release Doctor
Crawl a website before or after deployment, find technical release blockers, prepare exact fixes, and verify what changed without creating an account.
SEO & MarketingMeta Tag Generator
Create a clean title, description, canonical, robots, Open Graph and viewport tag block, ready to paste.
SEO & MarketingOpen Graph Generator
Generate Open Graph tags for rich previews on social and messaging apps.
SEO & MarketingRobots.txt Generator
Create a practical robots.txt file with crawl rules and sitemap location.
SEO & MarketingUTM Campaign URL Builder
Add consistent UTM campaign parameters to any destination URL.
SEO & MarketingSERP Snippet Preview
Preview a page title, URL, and description as a search result.
SEO & Marketing