Free Tool
Is your site blocking AI crawlers?
If your robots.txt blocks GPTBot, ClaudeBot, or PerplexityBot, you've opted out of AI search — often by accident. Check any domain in seconds.
Free · No account required · We only read your public robots.txt
We check the ten crawlers that feed the major AI assistants — including GPTBot, ClaudeBot, PerplexityBot, and Google-Extended — and tell you exactly which your site allows.
What this AI crawler check looks at
A crawler check reads your public robots.txt and works out, bot by bot, which AI crawlers are allowed to read your site. A crawler is reported as blocked only when the group governing it disallows your site root — a partial rule like Disallow: /admin doesn't count. These are the ten crawlers we test:
| Crawler | Company | What it feeds |
|---|---|---|
| GPTBot | OpenAI | Trains models + powers ChatGPT answers |
| OAI-SearchBot | OpenAI | ChatGPT search results |
| ChatGPT-User | OpenAI | User-triggered browsing in ChatGPT |
| ClaudeBot | Anthropic | Powers Claude |
| PerplexityBot | Perplexity | Perplexity answers + citations |
| Google-Extended | Gemini / AI Overviews training | |
| Applebot-Extended | Apple | Apple Intelligence |
| meta-externalagent | Meta | Meta AI |
| CCBot | Common Crawl | Feeds many open models |
| Bytespider | ByteDance | TikTok / Doubao AI |
How to unblock an AI crawler in robots.txt
If the check finds a crawler blocked and you want AI assistants to be able to read and cite your site, find the User-agent group naming that bot and remove its Disallow: / line — or delete the group entirely, so the bot falls back to your wildcard rules. A blocking rule looks like this:
User-agent: GPTBot
Disallow: /Blocking often happens by accident: a CDN or security plugin that ships an "AI bot blocklist", or a robots.txt template copied from another site. That's why it's worth re-running a crawler check after any deploy or plugin change — and why this tool offers a free weekly re-check.
AI crawler checker — common questions
- What is an AI crawler?
- A bot that reads web pages to feed AI systems — either to train models or to fetch pages that power live answers. Each identifies itself with a user-agent string (like GPTBot or ClaudeBot) that your robots.txt can allow or block.
- Does blocking AI crawlers affect my Google rankings?
- No. Googlebot is a separate crawler, so blocking Google-Extended or any AI bot doesn't change classic search rankings. What it changes is whether Gemini, ChatGPT, Perplexity, and the other assistants can read your content — and an assistant that can't read you is unlikely to cite you.
- Should I block or allow AI crawlers?
- If AI answers are a channel you want to show up in, allow them. If you're a publisher whose business depends on page views, blocking training bots is a legitimate choice. The failure mode is doing it by accident — which is exactly what this check catches.
- How often should I run a crawler check?
- After any deploy, CDN change, or plugin install that could touch robots.txt — or let us re-check weekly and email you only when something changes.
Crawler access is step one. Are you actually being cited?
Allowing the crawlers doesn't guarantee AI assistants mention you. See where you actually appear across ChatGPT, Gemini, Perplexity, and Claude.