Robots.txt Generator

Generate robots.txt for your website

FreeNo sign-upSEO Tools →

Robots.txt Generator writes a robots.txt file from rules you build in a form. Each rule targets a user agent, either all crawlers or a specific one such as Googlebot, Bingbot, DuckDuckBot, Baiduspider or YandexBot, and lists the paths to allow and disallow, one per line.

Add as many rule groups as you need, then set the sitemap URL and an optional crawl delay. The finished file is shown below and can be copied. Save it as robots.txt in the root of your domain so it is reachable at yourdomain.com/robots.txt.

Advertisement

How to use Robots.txt Generator

1

Add a rule

Pick a user agent and enter allow and disallow paths.

2

Add the sitemap

Enter the full URL of your XML sitemap.

3

Copy and upload

Save the output as robots.txt at the site root.

Use Cases

  • Keeping admin and cart URLs out of the crawl
  • Pointing crawlers to a new sitemap
  • Writing separate rules for one specific bot

Frequently Asked Questions