Two kinds of AI bot
AI companies run different crawlers for different jobs. Search and retrieval bots fetch pages so an assistant can cite them in an answer. Training bots collect content to train future models. Many businesses want to allow the first and choose freely about the second.
The main crawlers
| User agent | Company | Purpose |
|---|---|---|
| OAI-SearchBot | OpenAI | ChatGPT search results |
| ChatGPT-User | OpenAI | Pages fetched when a user asks |
| GPTBot | OpenAI | Model training |
| PerplexityBot | Perplexity | Perplexity’s search index |
| Claude-SearchBot / Claude-User | Anthropic | Claude search and user-requested fetches |
| ClaudeBot | Anthropic | Model training |
| Googlebot | Search, including AI Overviews and AI Mode | |
| Google-Extended | A control token for Gemini training and grounding; not a separate crawler | |
| Bingbot | Microsoft | Bing and Microsoft Copilot |
Bot names change. Check each company’s documentation before you finalise your rules.
A sensible robots.txt
This example allows search and retrieval bots and blocks training bots. Remove the training blocks if you are happy for your content to be used in training.
# Allow AI search and user-requested fetches
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
# Optional: opt out of model training
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
Sitemap: https://www.example.in/sitemap.xmlCheck for hidden blocks
- CDN settings. Cloudflare and other providers now offer AI-bot blocking; check yours is not blocking search bots by default.
- WordPress security and AI-bot plugins. Review their bot policies after every update.
- Hosting firewalls. Some block unfamiliar user agents automatically.
- JavaScript-only content. Not every AI crawler renders JavaScript; keep key text in the HTML.
What about llms.txt?
llms.txt is a proposed standard: a plain-text file at your site root that lists your most important pages for AI systems. It is quick to add and does no harm. But no major AI company has confirmed that it changes rankings or citations, so treat it as a tidy extra, not a strategy.
# AISEOAgency.in
> AI SEO agency in India offering GEO, AEO and technical SEO.
## Key pages
- [Services](https://aiseoagency.in/#services)
- [Pricing](https://aiseoagency.in/#pricing)
- [AI SEO in Bhopal](https://aiseoagency.in/ai-seo-agency-bhopal/)Quick checklist
- List which bots you allow and block, and why.
- Update robots.txt and test it with each search console.
- Check your CDN, firewall and plugins.
- Make sure key content is in the HTML.
- Add llms.txt if you like.
- Recheck every quarter.
Frequently asked questions
Will blocking GPTBot remove me from ChatGPT?
Not from ChatGPT search, which uses OAI-SearchBot. GPTBot is used for training. Allow OAI-SearchBot if you want to be cited.
Does blocking Google-Extended affect AI Overviews?
No. AI Overviews are part of Google Search, which uses Googlebot. Google-Extended controls use for Gemini model training and grounding.
Is llms.txt required?
No. It is optional, and its effect on AI citations is unproven.
