Free AI Crawler Checker (+ Guide)

What is an AI crawler?

An AI crawler is a bot that an AI company sends to read web pages. Some read pages to train AI models, some to answer searches, and some fetch 1 page because a person asked.

Most follow the rules in your robots.txt, the file at the root of your site that says which bots may read what.

How to check if your site blocks AI crawlers

Paste a page's full link to check that page's path instead of the home page.

The 3 kinds of AI crawlers

Each AI company runs different crawlers for different jobs. Blocking one does nothing to the others:

So you can block GPTBot to keep out of OpenAI's training, and still show up in ChatGPT search through OAI-SearchBot.

Tokens that aren't crawlers

Google-Extended and Applebot-Extended don't crawl anything. They're names in robots.txt that tell Google and Apple whether their normal crawlers' pages may train AI.

Blocking Google-Extended keeps your pages out of Gemini's training and grounding, according to Google's crawler docs. It doesn't touch Google Search.

Crawlers that may skip robots.txt

A fetch a person asked for is treated differently. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them.

Meta says the same of Meta-ExternalFetcher. The checker marks each of these, so you don't count on a rule they may not read.

How to allow AI search and block AI training

Pick "AI search and user fetches yes, training no" in the checker, and copy the lines it writes. They look like this:

robots.txt lines
User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

Put them in your robots.txt, at the root of your site. Then check your site again to see the change.

What the checker can't see

robots.txt isn't the only gate. A firewall, a CDN like Cloudflare or your host can block AI crawlers before they ever read your robots.txt.

The checker reads your site under its own name, never pretending to be GPTBot. So if your robots.txt allows a crawler but your firewall blocks it, check your host's bot settings too.

Your crawler access score

The score is out of 100, and it has two halves.

The first half is the search engines, Googlebot first. The second is AI search, with the fetches people ask an AI for, like ChatGPT opening a page you linked.

I test both on your home page and up to nine pages from your sitemap, so a rule that only blocks your blog still shows up.

Training crawlers never count. Keeping GPTBot or ClaudeBot out of training is a choice, and the score shouldn't punish it.

Here's what three sites scored on October 1, 2026:

The New York Times keeps most AI search crawlers out, so its AI half drops to 31 while Google still reads everything.

Every check in the report

Failed checks and warnings also go into a fix list at the top, with what to change for each.

Content signals in robots.txt

Content signals are a newer, optional line in robots.txt. Cloudflare published the policy on September 24, 2025.

The policy has one signal per use:

Vercel's robots.txt allows search and AI answers but says no to training. That's a clear way to say "cite me, don't train on me."

A signal doesn't block anyone, though. It states a preference, and the Disallow lines still decide who may crawl.

Limits Google sets on robots.txt

Google doesn't support crawl-delay, so the line does nothing for Googlebot. Bing and some other crawlers slow down when they see it.

Google also reads only the first 500 KiB of a robots.txt. Anything after that is ignored, rules included.

And a noindex line in robots.txt has done nothing since September 1, 2019, when Google stopped reading it. Use a noindex meta tag on the page instead.

When my checker can't load your sitemap within 12 seconds, it says so. A slow answer is a limit on my side, so it never counts as a failed check.

llms.txt and your sitemap

The checker also looks for an llms.txt, a short guide to your site for AI, and for your sitemap.

If you don't have an llms.txt, make one in a minute with my free llms.txt generator.