Most websites accidentally block AI crawlers by omitting them from robots.txt. If GPTBot, PerplexityBot, or ClaudeBot are not explicitly listed as allowed, they may not crawl your site — meaning you cannot appear in AI-generated answers regardless of how good your content is. This is the single fastest fix in AEO.
How to check if your site blocks AI crawlers
Open a browser and go to yourdomain.com/robots.txt. Look for the names of AI crawlers. If they are missing entirely, or if they appear with Disallow: /, you are blocked.
A wildcard User-agent: * Allow: / covers most bots but does not guarantee AI crawlers will respect it — explicitly listing each AI bot is the most reliable approach.
The exact robots.txt lines to add
Add these lines to your robots.txt. Each bot needs its own User-agent entry:
User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: PerplexityBot Allow: / User-agent: ClaudeBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Google-Extended Allow: / User-agent: bingbot Allow: /
Search bots vs training bots — what is the difference?
There are two types of AI crawlers with very different roles:
| Bot | Engine | Recommendation |
|---|---|---|
| GPTBot | ChatGPT | Powers ChatGPT search citations — never block |
| OAI-SearchBot | ChatGPT | OpenAI search crawler — never block |
| ChatGPT-User | ChatGPT | User-initiated browsing — never block |
| PerplexityBot | Perplexity | Powers Perplexity citations — never block |
| ClaudeBot | Claude | Powers Claude citations — never block |
| Claude-SearchBot | Claude | Claude search crawler — never block |
| Google-Extended | Google AI Overviews | AI training and SGE — allow for citations |
| bingbot | ChatGPT (via Bing) | Bing index drives ChatGPT — never block |
Blocking training bots (like CCBot or CommonCrawl) is a personal choice and has no effect on your live citation rate. Blocking search bots like GPTBot and PerplexityBot directly prevents you from appearing in those engines' answers.
Frequently asked questions
Does this fix work immediately?
AI bots check robots.txt before crawling. Once you update the file, bots see the new rules within 24-48 hours on their next crawl cycle. First citation improvements typically follow within 30-60 days.
Should I block AI training bots?
That's your choice — some site owners prefer to opt out of AI model training. Importantly, blocking training bots does not affect your live citation rate. The bots that power real-time answers (GPTBot search mode, OAI-SearchBot, PerplexityBot) are separate from training crawlers.
Run a free AEO scan
Our free scanner checks your robots.txt, schema markup, Bing indexing, and AI visibility in 60 seconds.
Free Scan at scan.affordableaeo.co