CCBot
CCBot collects pages for a public web dataset. It isn't an AI assistant that answers questions about businesses.
- Operated by
- Common Crawl
- robots.txt token
CCBot- User-Agent sent
- CCBot/2.0 (https://commoncrawl.org/faq/)
- Honours robots.txt
- Yes — Common Crawl documents that it honours robots.txt.
This one is not on the answer path.
CCBot collects pages for a public web dataset. It isn't an AI assistant that answers questions about businesses, and blocking it is a common, deliberate choice.
Whether to be included in a public web corpus is a judgement call about your content, not a visibility decision — so this page does not make a recommendation either way.
Two lines in robots.txt.
Block CCBot
User-agent: CCBot
Disallow: /Goes in the robots.txt at the root of your domain.
Allow CCBot
User-agent: CCBot
Allow: /Only needed if a broader rule already blocks it — a group of its own overrides the wildcard.
Group selection matches the token, not the full User-Agent string, and it is case-insensitive. A rule under User-agent: * applies only when the crawler has no group of its own.
Everything on this page was read from Common Crawl’s own published source on . Read the source. Vendors change these pages without announcing it — if you find something here that no longer matches, tell us and we’ll correct it.
We run a crawler too. Here is what it does and how to block it.
Knowing what CCBot does doesn’t tell you whether your site lets it in.
The free checker reads your robots.txt and asks your site for its homepage as each major AI crawler, then tells you which ones it turned away. No signup, about 20 seconds.