There is a version of the AI-visibility problem that no amount of content or PR can fix, because it happens before any of that matters: the AI assistants are not allowed to read your site at all.

It is more common than you would think. A security plugin tightens robots.txt. A CDN adds a "block AI bots" toggle that someone flips without thinking it through. An SEO tool "optimizes" your crawl directives. Any of these can quietly wall off GPTBot, ClaudeBot, PerplexityBot, and Google-Extended — and from that moment, those assistants are working from whatever they knew about you before, or from what other sites say, never from your own pages.

This guide covers how to check, what each bot actually does, and how to decide who to let in.

The two-minute check

The fastest way to know is to look. The free AI Crawler Access Checker reads your robots.txt and tells you which AI agents are allowed and which are blocked.

You can also check manually. Open https://yourdomain.com/robots.txt and look for blocks like:

User-agent: GPTBot
Disallow: /

A Disallow: / under an AI user-agent means that bot is shut out of your entire site. An empty Disallow: or no rule at all means it is allowed.

Who the main AI crawlers are

Not all bots do the same thing, and that distinction matters for the decision below.

  • GPTBot (OpenAI). Crawls content that can inform ChatGPT, including for retrieval. Blocking it reduces your odds of being read and cited by ChatGPT.
  • OAI-SearchBot (OpenAI). Associated with ChatGPT's search/browse features — the live-retrieval path, distinct from training crawls.
  • ClaudeBot (Anthropic). Crawls content that can inform Claude.
  • PerplexityBot (Perplexity). Perplexity leans heavily on live retrieval and citation, so access here is especially tied to whether you get cited.
  • Google-Extended (Google). Controls whether your content can be used for Google's generative AI features (like AI Overviews and Gemini), separately from normal Google Search crawling. Blocking Google-Extended does not remove you from Google Search — but it can remove you from Google's AI answers.

The key nuance: these are mostly separate from your normal search crawlers. You can be perfectly indexed in Google Search and still be invisible to the AI layer because Google-Extended or GPTBot is blocked.

Should you actually let them in?

For most brands that want to be found and recommended, yes — allowing the major assistant crawlers is how you become eligible to appear in their answers. If your buyers are asking ChatGPT and Perplexity about your category, being unreadable there is lost consideration you cannot see in any dashboard.

But it is a genuine decision, not a reflex. Reasons a site might restrict some bots:

  • Proprietary or paywalled content you do not want summarized for free.
  • Bandwidth or cost concerns from heavy crawling on very large sites.
  • A deliberate stance on how your content is used for AI training.

A reasonable middle path many sites take: allow the retrieval- and citation-oriented bots (so you can be cited in live answers) while making a separate, deliberate choice about training-oriented crawls. The point is to decide on purpose — not to discover months later that a plugin made the decision for you.

How to unblock them

If you find AI bots blocked and you want them in, edit your robots.txt to allow the agents you have chosen. For example, to explicitly allow the main ones:

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

Then re-check to confirm the change took effect, and remember that some assistants only re-crawl periodically — access is the prerequisite, not an instant result.

Access is necessary, not sufficient

Unblocking the crawlers puts you back in the running. It does not, by itself, get you named in AI answers — that still depends on answering real questions directly, being corroborated across third-party sources, and being a clear, consistent entity. See How to Rank in ChatGPT for those levers, and The Complete Guide to AI Search Visibility for the full picture.

But think of it this way: everything else on the AI-visibility checklist is you trying to earn a spot in the answer. Crawler access is the one where you can accidentally disqualify yourself before the game starts. It is worth two minutes to make sure you have not.

For agencies: check every client

If you manage a roster of clients, AI crawler access is a fast, high-value audit to run across all of them — it is common, it is invisible until you look, and finding a blocked bot is a concrete, easy win you can bring to a client conversation. Run each client's site through the AI Crawler Access Checker, and fold ongoing AI visibility into the reporting you already deliver. GEO for Agencies covers how to make it a service rather than a one-off.

Start a free Trafiq trial to track AI visibility for every client workspace — starting with whether the assistants can even read the site.