Are the AI search bots that cite your pages allowed to crawl your site?
AI search crawlers must reach your pages before they can ever cite you in answers from ChatGPT, Perplexity, or Bing.
What this signal tests
We check whether the AI-search crawlers - the bots that power ChatGPT Search, Perplexity, Claude's web tools, Bing Copilot, and Apple Intelligence citations - are permitted to crawl your site. Specifically we make sure your robots.txt does not contain a blanket Disallow rule that shuts them out of the entire domain.
Why it matters for your visibility in AI
AI-search bots are different from AI training bots: they exist to fetch pages on demand when a user asks the AI assistant a question, and then to cite or quote the result. If you block them, your site cannot appear in any AI-generated answer, period. No amount of great content or schema markup compensates for the door being closed. This is one of the most damaging configuration mistakes we see. Sites that block AI search bots - often by accident, copied from old privacy-focused robots.txt templates - disappear from the answer interfaces that an increasing share of customers now use instead of Google. Meanwhile competitors are quoted and linked, and over time the brand becomes invisible in AI conversations.
Pass criteria at a glance
| Criterion | Passes when |
|---|---|
| Zero blanket Disallow on these search bots. |
How we test it
We simulate each AI-search bot - OAI-SearchBot, PerplexityBot, Claude-SearchBot, Bingbot, Applebot, and Google-CloudVertexBot - visiting your homepage and a few sample URLs from your sitemap. We apply the same matching logic these bots use when reading your robots.txt, and we flag any case where a bot would be blocked at the root or across the entire site by a blanket Disallow rule.
Show technical detection method
Simulate RFC 9309 path matching for each search bot against `/` and sample sitemap URLs; flag any blanket Disallow.
If your site fails: how to fix it
- Open robots.txt and find any User-agent block that contains the line Disallow followed by a single slash - that pattern blocks the entire site.
- For each AI-search bot you want indexed (OAI-SearchBot, PerplexityBot, Claude-SearchBot, Bingbot, Applebot, Google-CloudVertexBot), either remove the blanket Disallow rule or add an explicit Allow rule for the root path.
- If you legitimately need to keep some areas private - for example /admin/ or /checkout/ - replace the blanket block with targeted Disallow rules pointing at those specific paths.
- Double-check the wildcard User-agent block too: if it contains a blanket Disallow, that rule will apply to any bot not named explicitly.
- Re-run the AI Ready Test and confirm each search bot is now able to reach your homepage and your sitemap URLs.
Quick facts
| Maturity | ESTABLISHED |
|---|---|
| Weight | high |
| Category | Crawlability |
Primary sources
Related signals
Frequently asked questions
How is an AI-search bot different from an AI training bot?
Search bots fetch a page in real time when a user asks the AI a question and may then cite or quote it. Training bots crawl bulk content to be used in future model training. Many sites are comfortable allowing the first but not the second - these are separately controllable.
If I block training bots, do I block search bots too?
No - they are different user agents and can be configured independently. For example, you can disallow GPTBot (training) while allowing OAI-SearchBot (real-time search). This is the most common modern policy because it preserves AI visibility without consenting to model training.
How quickly will I be back in AI answers after fixing this?
Once the block is removed, search bots can crawl on their next visit, often within hours. Re-appearing in actual AI-generated answers depends on how frequently each assistant rebuilds its citation pool - typically days to weeks for routine queries.
What if I want my site visible in ChatGPT but not in Bing Copilot?
You can target each bot individually: allow OAI-SearchBot and disallow Bingbot in robots.txt, for example. Be aware though that Google AI Overviews and Bing Copilot share much of their crawl infrastructure with the underlying classic search engines, so blocking them affects regular search too.
Run your own scan
Run a free scan and see how your site grades across all 155 AI-readiness signals.