Is your robots.txt file readable by ChatGPT and other AI crawlers?
A valid robots.txt at your site root tells AI crawlers what they can read and index.
What this signal tests
We check whether your site publishes a robots.txt file at the standard location, whether it loads successfully, and whether its contents follow the official rules that AI crawlers expect. A robots.txt file is the rulebook every search and AI bot reads first before crawling any other page on your domain.
Why it matters for your visibility in AI
If your robots.txt file is missing, broken, or returns an error, well-behaved AI crawlers from OpenAI, Anthropic, Google, and Perplexity assume the worst and treat the entire site as off-limits. Your pages then quietly disappear from ChatGPT Search, Claude, Perplexity, and Google AI Overviews. This is the most common reason a site is invisible to AI assistants despite ranking well on traditional Google. A competitor with a working robots.txt gets cited in answers about your industry, while your brand is never mentioned, simply because a single file at the root of your site cannot be read.
Pass criteria at a glance
| Criterion | Passes when |
|---|---|
| 200 + valid grammar + size limit met. |
How we test it
We request the file at /robots.txt over HTTPS and confirm three things. First, the server returns a successful response. Second, the file is served as plain text in a standard character encoding and is not absurdly large. Third, the contents parse cleanly against the published internet standard for robots files (RFC 9309), with no syntax errors that would cause a bot to give up reading it.
Show technical detection method
GET https://{host}/robots.txt. Assert 200 + text/plain + UTF-8 + size<=500KiB + zero RFC 9309 parse errors.
If your site fails: how to fix it
- Create a plain text file called robots.txt at the root of your web server, so it loads at https://yourdomain.com/robots.txt. Most CMSs and hosting panels have a setting for this; otherwise add the file to your public directory.
- Add a minimal permissive rule set as a starting point: a User-agent line of asterisk followed by a Disallow line that is left empty. This allows all bots to crawl everything while keeping the file valid.
- Save the file as UTF-8 encoded plain text and keep it under 500 kilobytes. Avoid HTML, BOM markers, or smart quotes from word processors - these break parsing.
- Confirm your server returns it with the content type text/plain and a 200 status code. Some frameworks accidentally serve robots.txt as HTML or 404 when the file is missing.
- Re-run the AI Ready Test to confirm the file is now detected and parses cleanly.
Quick facts
| Maturity | ESTABLISHED |
|---|---|
| Weight | high |
| Category | Crawlability |
Primary sources
Related signals
Frequently asked questions
How long does this take to fix?
For most sites, less than 15 minutes. Creating and uploading a small text file is one of the simplest changes you can make to a website, and the effect is immediate - the next time an AI crawler visits your domain, it will see the new file.
Does this affect my Google search ranking too?
Yes. The same robots.txt file is read by Googlebot, Bingbot, and every other compliant crawler. A broken or missing robots.txt can suppress traditional search rankings as well, so fixing this is a win for both classic SEO and AI visibility.
What if I do not want any bots to read my site?
You should still publish a valid robots.txt - one that explicitly disallows them. The file itself must exist and parse correctly. A missing file is ambiguous; an explicit Disallow rule is a clear policy that compliant bots will respect.
Where do I put the file if my site uses a CMS like WordPress or Shopify?
Most platforms generate a robots.txt automatically. WordPress lets you override it via SEO plugins such as Yoast or Rank Math. Shopify lets you edit it through the robots.txt.liquid template. Check your platform's documentation for the exact location.
Run your own scan
Run a free scan and see how your site grades across all 155 AI-readiness signals.