The AI crawlers, one page each.
What each one is for, whether its vendor says it honours robots.txt, what blocking it actually costs you, and the exact two lines to do it. Every fact read from the vendor’s own documentation, with the date we read it.
A vendor’s bots are not one thing.
OpenAI alone runs three, and they do different jobs. One collects content to train models. One builds the index ChatGPT answers from. One opens a page in the moment because somebody asked about it. Blocking the first does not block the other two — and only the last two change what an assistant can say about you today.
Almost every guide online flattens that into one rule. It is why “should I block GPTBot” has no single answer, and why each of these gets its own page.
OpenAI
- OAI-SearchBotAI answer path
OpenAI uses OAI-SearchBot to index sites for ChatGPT's search answers.
- ChatGPT-UserAI answer path
ChatGPT uses ChatGPT-User to open a page when someone asks a question about it.
- GPTBotModel training
GPTBot collects content for training OpenAI's models. It isn't what ChatGPT uses to find businesses.
Anthropic
- Claude-SearchBotAI answer path
Anthropic uses Claude-SearchBot to index sites for Claude's search results.
- Claude-UserAI answer path
Claude uses Claude-User to open a page when someone asks a question about it.
- ClaudeBotModel training
ClaudeBot collects content for training Anthropic's models. Claude uses separate crawlers to answer questions.
Perplexity
Which of these is your site turning away?
The free checker reads your robots.txt and asks your site for its homepage as each major AI crawler, then tells you in plain English which ones it refused. No signup, about 20 seconds.