Robots.txt Tester

Test robots.txt rules against URLs

FreeNo sign-upSEO Tools →

Robots.txt Tester tells you whether a URL path is allowed or blocked by a robots.txt file. Paste the file contents, enter a path such as /admin/page, choose a user agent, and it reports the verdict and the rule that decided it.

Matching follows the rules Google documents: the most specific user-agent group applies, the longest matching path wins, and Allow wins a tie. The * wildcard and the $ end-of-URL anchor are supported. Test changes here before uploading them, because a mistaken Disallow can remove a whole section from search.

Advertisement

How to use Robots.txt Tester

1

Paste your robots.txt

The whole file.

2

Enter a path and user agent

For example /blog/post and Googlebot.

3

Read the result

Allowed or blocked, with the matching rule.

Use Cases

  • Confirming a new Disallow does not block key pages
  • Debugging a page reported as blocked in Search Console
  • Checking wildcard rules before deployment

Frequently Asked Questions