ClaudeBot
ClaudeBot collects content for training Anthropic's models. Claude uses separate crawlers to answer questions.
- Operated by
- Anthropic
- robots.txt token
ClaudeBot- User-Agent sent
- Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)
- Honours robots.txt
- Yes — Anthropic documents that it honours robots.txt.
Anthropic documents the token but did not publish a full User-Agent string on the page we read. The string above is the one reported consistently elsewhere — treat the token, not the string, as the part to match on.
This one is not on the answer path.
ClaudeBot collects content for training Anthropic's models. Claude uses separate crawlers to answer questions, so this doesn't affect whether you show up in Claude's answers.
Whether to allow model training is a judgement call about your content, not a visibility decision — so this page does not make a recommendation either way.
Two lines in robots.txt.
Block ClaudeBot
User-agent: ClaudeBot
Disallow: /Goes in the robots.txt at the root of your domain.
Allow ClaudeBot
User-agent: ClaudeBot
Allow: /Only needed if a broader rule already blocks it — a group of its own overrides the wildcard.
Group selection matches the token, not the full User-Agent string, and it is case-insensitive. A rule under User-agent: * applies only when the crawler has no group of its own.
Anthropic runs more than one, and they do different jobs.
Blocking one does not block the others. This is where most advice goes wrong: the rule that stops model training is not the rule that decides whether you appear in an answer.
Everything on this page was read from Anthropic’s own published source on . Read the source. Vendors change these pages without announcing it — if you find something here that no longer matches, tell us and we’ll correct it.
We run a crawler too. Here is what it does and how to block it.
Knowing what ClaudeBot does doesn’t tell you whether your site lets it in.
The free checker reads your robots.txt and asks your site for its homepage as each major AI crawler, then tells you which ones it turned away. No signup, about 20 seconds.