Free Robots.txt Generator (+ Guide)

What is a robots.txt file?

A robots.txt file is a plain text file at yoursite.com/robots.txt that gives crawlers rules.

Each group of rules names a crawler, then lists the paths it may or may not read.

Google's own introduction to robots.txt says it's mainly there to stop crawlers from overloading your site... and that it's not a way to keep a page out of Google.

It's also a request, not a lock. Googlebot and the big AI companies follow it, but a scraper can ignore it completely.

What the robots.txt generator does

It also warns you about the mistakes that cost the most, like keeping Google out of every page or blocking your styles and scripts.

How to make a robots.txt file

The four lines Google reads

Google's robots.txt spec supports four fields, and ignores the rest:

Crawl-delay isn't on the list, so Google ignores it (Bing reads it). And Google stopped reading noindex in robots.txt back in 2019.

One small detail catches people out: paths are case-sensitive, so /Blog/ and /blog/ are different.

Crawler names aren't, so GPTBot and gptbot are the same crawler.

How Google picks the rule

This is the part that trips people up, so here's how it works, step by step.

First, a crawler looks for the group that names it. If there's one, it follows only that group and ignores the * group completely.

Then, inside its group, the most specific rule wins, which means the one with the longest path. And if an Allow and a Disallow are equally long, Google picks the least restrictive one, the Allow.

You can also use two wildcards: * matches anything, and $ marks the end of the address. So Disallow: /*.pdf$ keeps crawlers out of every PDF.

The generator writes the file with these rules in mind, and the test reads it the same way, following the standard, RFC 9309.

What to keep out

These are the paths I'd keep crawlers out of on most sites:

• Admin and login pages, where nothing helps a searcher
• Cart and checkout pages, which are different for every visitor
• Site search results, which can create endless pages with little value
• Thank-you pages after a sign-up or a purchase
• Your API, if your site has one

The presets add the common ones in one click, like /wp-admin/ with an Allow for admin-ajax.php, which WordPress themes need.

AI crawlers: search and training are different

This is where robots.txt changed the most in the last few years.

Most AI companies now run separate crawlers for separate jobs, and each one has its own name in robots.txt:

So you can keep GPTBot out of training and still let OAI-SearchBot put your pages in ChatGPT's answers.

Google-Extended is a special case. Google says it controls whether your pages train Gemini and ground its answers, but it doesn't affect your place in Google Search at all, and it isn't a ranking signal.

The AI crawler checker shows what each of these crawlers may read on any site today.

A real example: my robots.txt and the New York Times

I tested an article address on two very different sites, against the same 23 crawlers.

My own robots.txt lets every crawler read my articles, AI training included. It keeps all of them out of the same few paths, like my API and a couple of thank-you pages.

The New York Times goes the other way. It keeps out 15 of the 23, including OpenAI's, Anthropic's and Perplexity's AI search crawlers... while Googlebot can still read everything.

Neither one is wrong. It's a choice, and the generator makes it a few clicks instead of hundreds of lines of text.

Where the file goes

The file only works at the root of your site, like example.com/robots.txt. Crawlers don't look anywhere else.

It also only covers the host it sits on. So blog.example.com needs its own robots.txt, and so does a different protocol or port.

And two limits from Google: it reads up to 500 KiB of the file and ignores anything after that, and it usually keeps a copy for up to 24 hours. So a change can take a day to show.

robots.txt doesn't remove a page from Google

This one's easy to miss.

Google says a page blocked by robots.txt can still show in search results, just without a description, if other sites link to it.

So to keep a page out of search, add a noindex robots meta tag and let Google read the page. If Google can't read the page, it never sees the noindex.

Mistakes to avoid

• Disallow: / for every crawler on a live site, which keeps Google out of everything
• Blocking the folders with your styles and scripts, so Google can't see pages the way people do
• Using robots.txt to hide private pages, when anyone can read the file
• Blocking a page you want removed from Google, so it never sees the noindex
• Keeping out every AI crawler when you only meant to stop training
• Forgetting a subdomain, which needs its own file

Who is the robots.txt generator for?

It's for anyone who runs a site and wants to decide who reads it:

• How to start a blog – bloggers setting up a new site
• How to create an online course – course creators keeping checkout and lesson pages out of search
• AI crawler checker – anyone deciding what AI companies may read
• Sitemap generator – site owners who want crawlers to find every page
• llms.txt generator – site owners who want AI tools to understand their site

And if you've never looked at your robots.txt, test it once with the robots.txt tester. It takes a few seconds.

After you make your robots.txt