robots.txt checker
See who your site lets in — exactly which AI crawlers like GPTBot, ClaudeBot, Google-Extended and PerplexityBot are allowed or blocked — then test what a crawler actually reads.
Test if a crawler can scrape your page
Being “kept out” is only a request.
robots.txt asks bots to stay out — it can’t stop anyone, and it doesn’t guarantee the ones you allow can even read the page. See what a crawler really gets:
How to check robots.txt for AI crawlers
Any website can publish a robots.txt file at its root that tells crawlers which parts of the site they may request. This checker reads that file and sorts the result by AI crawler — so you can see at a glance whether bots like GPTBot, ClaudeBot and Google-Extended are allowed in or kept out.
Use this checker Fastest
- 1Paste any address into the box at the top of this page — a bare domain like
example.com, a deep link, or a full URL all work. - 2Click Check robots.txt. We fetch the file from the site’s origin root and parse it in seconds, with no signup.
- 3Read the per-crawler verdict: each AI bot is sorted into Allowed in or Kept out, alongside the default rule, other named crawlers and the raw file.
robots.txt is read per-origin at the host root, so the address you paste is resolved to {scheme}://{host}/robots.txt — a link to example.com/blog/post is checked at example.com/robots.txt. Subdomains are treated as given: blog.example.com is checked at blog.example.com/robots.txt, not the apex domain.
Read it by hand For the technically-minded
You can also read any site’s robots.txt directly. Request the file from the origin root with curl:
curl https://example.com/robots.txt
The file is a list of groups. Each group opens with one or more User-agent lines naming the crawler it applies to, followed by rule lines:
User-agent: — names the crawler a group of rules applies to. User-agent: * is the default group for any crawler not named elsewhere in the file.
Disallow: — asks the crawler not to request paths that start with the given value. Disallow: / asks it to stay off the whole site; an empty Disallow: blocks nothing.
Allow: — carves an exception out of a broader Disallow, letting a crawler request a path that would otherwise be blocked.
Reading it by hand is fine for a short file, but a busy robots.txt can name a dozen crawlers across overlapping groups — which is exactly what this checker untangles for the AI bots that matter.
The AI crawlers this tool checks for
Each identifies itself with a user-agent token. To ask one to stay out, add its group to your robots.txt.
Fetches a page when a ChatGPT user follows a link or asks about it.
User-agent: ChatGPT-User
Disallow: /
Learn more
Builds the search index behind ChatGPT search.
User-agent: OAI-SearchBot
Disallow: /
Learn more
Trains Gemini and grounds Google’s AI answers. Separate from Search (see below).
User-agent: Google-Extended
Disallow: /
Learn more
Powers answers in the Perplexity answer engine.
User-agent: PerplexityBot
Disallow: /
Learn more
Builds the open Common Crawl dataset that many LLMs train on.
User-agent: CCBot
Disallow: /
Learn more
Trains ByteDance’s Doubao and TikTok AI.
User-agent: Bytespider
Disallow: /
Trains Apple Intelligence.
User-agent: Applebot-Extended
Disallow: /
Learn more
Trains Meta’s Llama models and Meta AI.
User-agent: Meta-ExternalAgent
Disallow: /
Learn more
robots.txt is a request, not a lock.
robots.txt is an honour-system directive. It asks crawlers to stay out of certain paths — and well-behaved ones, including the major AI crawlers listed above, do honour it. But nothing in the file enforces anything. A crawler that ignores it faces no technical barrier; it simply requests the page anyway.
That is exactly why this page includes a live test-scrape. Blocking a bot in robots.txt tells you what a compliant crawler is asked to do; the test-scrape shows what any crawler can still pull off the page regardless. The gap between the two is the whole point.
So treat the two questions separately. To actually prevent access you need enforcement — a firewall or WAF rule that blocks the request at the edge. To make your content usable by the AI you do allow, you need it crawlable and clean — which is half of what makes an anti-hallucination guarantee possible in the first place.
Want an AI that reads your site the right way?
Resolve247 crawls your site, trains on the clean content, and answers your customers 24/7 — with citations and no hallucinations. Turn the pages you allow into a support agent that never makes things up.
Start a Free Trial30-day free trial. No credit card required.
robots.txt checker FAQ
A robots.txt checker reads a site's /robots.txt file and shows which crawlers it allows or blocks. This one adds an AI-crawler lens: it breaks the result down by AI bot (GPTBot, ClaudeBot, Google-Extended, PerplexityBot and more), not just Googlebot.
GPTBot, ChatGPT-User and OAI-SearchBot (OpenAI), Google-Extended (Google), ClaudeBot (Anthropic), PerplexityBot, CCBot (Common Crawl), Bytespider, Amazonbot, Applebot-Extended and Meta-ExternalAgent — plus Googlebot and Bingbot for contrast.
No. robots.txt is a request that well-behaved crawlers honour; it isn't enforcement. That's why this page includes a test-scrape — to show what a crawler can still read regardless of the file. To actually block access you need a firewall or WAF rule.
Add a group to your robots.txt: User-agent: GPTBot then Disallow: /. Repeat per bot, or use User-agent: * to ask all crawlers to stay out. Remember this is advisory (see above).
The site doesn't publish a robots.txt file at its root. With no file, every crawler is unrestricted by default. Whether that's what you want depends on your goals — the checker just reports the state.
No. Google-Extended (AI training) is separate from Googlebot (Search). Blocking Google-Extended keeps your content out of AI training/grounding while leaving Search indexing untouched.
Free, no signup, and nothing is stored — the robots.txt is fetched and parsed in the same request.