Bytespider
Bytespider is ByteDance's web crawler. ByteDance doesn't say what it collects pages for, and it isn't an AI assistant that answers questions about businesses.
- Operated by
- ByteDance
- robots.txt token
Bytespider- User-Agent sent
- Bytespider
- Honours robots.txt
- Not documented. ByteDance does not say, so a Disallow may or may not be honoured.
ByteDance publishes no documentation for this crawler — not the token, not a User-Agent string. The name above is the one the crawler sends; treat it as observed, not confirmed.
This one is not on the answer path.
ByteDance doesn't say what Bytespider collects pages for, but it isn't an AI assistant that answers questions about businesses, and blocking it is a common, deliberate choice.
ByteDance does not say what this crawler is for, so whether to allow it is a judgement call about your content, not a visibility decision — and this page does not make a recommendation either way.
Two lines in robots.txt.
Block Bytespider
User-agent: Bytespider
Disallow: /Goes in the robots.txt at the root of your domain.
Allow Bytespider
User-agent: Bytespider
Allow: /Only needed if a broader rule already blocks it — a group of its own overrides the wildcard.
Group selection matches the token, not the full User-Agent string, and it is case-insensitive. A rule under User-agent: * applies only when the crawler has no group of its own.
ByteDance publishes no documentation for Bytespider that we could find, as of . Nothing on this page is a ByteDance statement: it is the crawler’s name as it arrives, and what robots.txt does with that name. If ByteDance publishes something, tell us and we’ll update this page.
We run a crawler too. Here is what it does and how to block it.
Knowing what Bytespider does doesn’t tell you whether your site lets it in.
The free checker reads your robots.txt and asks your site for its homepage as each major AI crawler, then tells you which ones it turned away. No signup, about 20 seconds.