Bytespider
Bytespider collects pages for a public web dataset. It isn't an AI assistant that answers questions about businesses.
- Operated by
- ByteDance
- robots.txt token
Bytespider- User-Agent sent
- Bytespider
- Honours robots.txt
- Not documented. ByteDance does not say, so a Disallow may or may not be honoured.
ByteDance documents the token but did not publish a full User-Agent string on the page we read. The string above is the one reported consistently elsewhere — treat the token, not the string, as the part to match on.
This one is not on the answer path.
Bytespider collects pages for a public web dataset. It isn't an AI assistant that answers questions about businesses, and blocking it is a common, deliberate choice.
Whether to be included in a public web corpus is a judgement call about your content, not a visibility decision — so this page does not make a recommendation either way.
Two lines in robots.txt.
Block Bytespider
User-agent: Bytespider
Disallow: /Goes in the robots.txt at the root of your domain.
Allow Bytespider
User-agent: Bytespider
Allow: /Only needed if a broader rule already blocks it — a group of its own overrides the wildcard.
Group selection matches the token, not the full User-Agent string, and it is case-insensitive. A rule under User-agent: * applies only when the crawler has no group of its own.
Everything on this page was read from ByteDance’s own published source on . Read the source. Vendors change these pages without announcing it — if you find something here that no longer matches, tell us and we’ll correct it.
ByteDance does not publish a crawler policy we could verify for this agent, so this page is sourced less firmly than the others in this reference — treat the robots.txt behaviour above as unconfirmed.
We run a crawler too. Here is what it does and how to block it.
Knowing what Bytespider does doesn’t tell you whether your site lets it in.
The free checker reads your robots.txt and asks your site for its homepage as each major AI crawler, then tells you which ones it turned away. No signup, about 20 seconds.