Free robots.txt tester and rule checker

What does a robots.txt tester check?

This free robots.txt tester evaluates a supplied rules file for one crawler and one URL path. It makes matching allow and disallow rules visible, highlights conflicts, and lets you export a readable file before you publish it.

A robots.txt file is a crawl-control document for a specific host, protocol, and port. It belongs at the top level—for example, https://example.com/robots.txt—and its path rules are relative to that origin.

How to use the robots.txt rule checker

1. Paste the exact robots.txt content you plan to serve.
2. Choose the crawler you want to evaluate, such as Googlebot.
3. Enter the complete path you need to test, including its leading slash.
4. Review the result, then verify the deployed file at /robots.txt and check important URLs separately.

User-agent: *
Disallow: /private/
Allow: /private/public-guide/
Sitemap: https://example.com/sitemap.xml

robots.txt is not a security or noindex control

Robots.txt asks compliant crawlers not to fetch matching paths; it does not protect private information, and a blocked URL can still be discovered or appear without a snippet. Use authentication for private content. When a crawlable response must stay out of search, use a valid noindex directive or another appropriate removal method.

Read the full robots.txt versus noindex guide for the decision and verification steps.

Common robots.txt mistakes

• Testing the wrong host: a subdomain has its own robots.txt scope.
• Using a folder-level file instead of the required top-level /robots.txt.
• Blocking CSS or JavaScript that a crawler needs to understand the page.
• Expecting a disallow rule to remove an already-known URL from search.

Method

Test robots.txt rules for crawlers and URLs, resolve allow/disallow conflicts, and export a clear file—free, private, and no account.

Privacy

Runs in the browser unless you explicitly load a public robots.txt file.

Limitations

Robots rules control crawling, not authorization, and do not guarantee deindexing.