Free Robots.txt Validator (+ Guide)

What is a robots.txt validator?

A robots.txt validator is a tool that checks a robots.txt file for mistakes: lines crawlers can't read, rules that don't do what they seem to, and fields Google ignores.

A tester tells you what a file allows, and a validator tells you if the file is right. This one does both.

How to validate your robots.txt

What the robots.txt validator checks

Every check follows Google's robots.txt rules, updated August 31, 2026, and the standard, RFC 9309.

It also lists every check the file passed, so you know what's already right.

A real example: a file with six mistakes

Press "Try a file with mistakes" on the tool to load this short file. I wrote it with six of the mistakes I see most often.

robots.txt
User-agent: *
Disallow: /cart/
Disallow: /Search
Dissallow: /checkout/
Crawl-delay: 10
Noindex: /old-page/

User-agent: Googlebot
Allow: /blog/

Sitemap: /sitemap.xml

Here's what the validator finds, worst first:

Line 8 is the one that hurts most. The file looks like it keeps every crawler out of the cart, but Googlebot follows only its own group, so it reads /cart/ anyway.

The most common robots.txt mistakes

These are the mistakes the validator catches most, and each one is a quick fix:

• A typo in a field name, like Dissallow, so Google skips the rule
• A crawler's own group without the * rules, so it reads what other crawlers can't
• A path without / at the start, which crawlers ignore
• A full link as a path, like Disallow: https://example.com/private/
• A Sitemap line with only a path, like Sitemap: /sitemap.xml
• Noindex or Crawl-delay, which Google doesn't read
• Blocking styles and scripts, so Google can't see pages the way people do
• The file in a folder, like /blog/robots.txt, where crawlers never look

Fix the errors first. A warning can wait a day, but an error means a rule isn't working right now.

Check your sitemap's pages against robots.txt

When you read a site, the validator also reads its sitemap, and fills in up to 500 of its pages as the addresses to check.

A sitemap lists the pages you want in Google, and robots.txt lists the ones crawlers should skip. So a page in both sends Google two opposite signals.

The validator warns you when Googlebot is kept out of any page in your sitemap. You can also pick any of the 23 crawlers, or all of them at once, and download the results as a .csv.

Real files from four big sites

I read four sites' robots.txt files with the validator on October 2, 2026:

None of them has an error. The New York Times gets six warnings, though, because Googlebot and five other crawlers have their own groups that skip some rules for *.

So Googlebot may read checkout pages that the other crawlers can't.

GitHub's 294 tips are almost all a * at the end of a path, which changes nothing. So a long list of tips doesn't mean a bad file.

How Google reads a robots.txt file

Google's rules for the file itself are short, and each one can cost you:

So a robots.txt that errors can be worse than having none at all.

Validator, tester or generator?

All three read robots.txt the same way, so pick the one for your job:

The tabs at the top of the tool switch between them, and keep the site you typed.

How I keep my robots.txt right

Who is the robots.txt validator for?

It's for anyone who edits a robots.txt, or wonders why Google skips a page:

• How to start a blog – bloggers checking a new site lets Google in
• AI crawler checker – anyone deciding what AI companies may read
• Sitemap generator – anyone whose file has no Sitemap line yet
• SEO checker – site owners who want every other check on a page

And if a page suddenly dropped out of Google, robots.txt is one of the first files I'd validate.