Robots.txt Generator writes a robots.txt file from rules you build in a form. Each rule targets a user agent, either all crawlers or a specific one such as Googlebot, Bingbot, DuckDuckBot, Baiduspider or YandexBot, and lists the paths to allow and disallow, one per line.
Add as many rule groups as you need, then set the sitemap URL and an optional crawl delay. The finished file is shown below and can be copied. Save it as robots.txt in the root of your domain so it is reachable at yourdomain.com/robots.txt.
Advertisement
How to use Robots.txt Generator
Add a rule
Pick a user agent and enter allow and disallow paths.
Add the sitemap
Enter the full URL of your XML sitemap.
Copy and upload
Save the output as robots.txt at the site root.
Use Cases
- Keeping admin and cart URLs out of the crawl
- Pointing crawlers to a new sitemap
- Writing separate rules for one specific bot
Frequently Asked Questions
Related Guides from Our Blog
Technical SEO Guide: Everything You Need to Know
A complete technical SEO guide covering crawlability, indexing, page speed, schema markup, sitemaps, robots.txt, and every foundational signal that affects your Google rankings.
Schema Markup Guide: How to Add Structured Data to Your Website
Schema markup tells Google exactly what your content means — products, FAQs, how-to guides, articles. This guide covers every major schema type with JSON-LD examples.
XML Sitemap Guide: Create, Optimise, and Submit Your Sitemap
An XML sitemap tells Google which pages exist on your site and when they were last updated. Here's how to create one correctly, optimise it, and submit it to Google.