Bytespider
Also known as ByteDance Bytespider
4 min read · updated 2026-09-09
Definition
Bytespider is ByteDance's crawler for collecting training data. ByteDance's training crawler. ByteDance is gathering training data.
For the reference card — Bytespider in the crawler directory.
What Bytespider is
Bytespider has the worst compliance reputation of any crawler on this site. Site owners consistently report it fetching pages that robots.txt disallows, and ByteDance publishes nothing that would let you verify or contest what you are seeing. If you need it stopped rather than merely asked to stop, that has to happen at the network layer.
ByteDance publishes no crawler documentation and no address ranges, and its main crawler has a long-standing reputation for ignoring robots.txt. What is known about these tokens comes from site owners' logs rather than from the company.
On this site Bytespider is counted as training, which is the row it appears in on your dashboard. The category matters because it decides which number moves when this crawler visits, and the four are reported separately for exactly that reason.
What its traffic looks like
Heavy, and frequently reported hitting pages that robots.txt disallows — including paths no link points at. If the volume looks disproportionate to your site's size, that matches what other site owners report.
How it is verified
ByteDance publishes neither a machine-readable list of the addresses this crawler uses nor a reverse DNS record pointing back at itself. The user-agent header is therefore the only evidence, and anyone can send it — so a visit is recorded as an unverified claim rather than as a confirmed fetch.
That is not a gap in the measurement so much as an honest ceiling. A report that showed these as verified would be inventing a confidence nobody can support, and the number it produced would be the first thing to fall apart when somebody asked where it came from.
If you block it
In principle, ByteDance stops collecting training data. In practice Bytespider is widely measured ignoring robots.txt, so a rule alone may not stop it and a firewall or WAF rule is the reliable option.
Nothing you see today changes if you block this. Training crawls feed models that ship months later, so the decision is about whether your work becomes part of a commercial model, not about this quarter's traffic.
The honest counter-argument is that being in the training data is part of how a model comes to mention you at all. There is no measurement that settles this either way — anyone who tells you there is has not looked — so it is a decision about your terms rather than about your numbers.
How it differs from ByteDance's other crawlers
ByteDance sends 3 crawlers and gives each its own token, so they can be allowed and refused separately. The tokens differ by a few characters and what the fetch means does not:
TikTokSpider — ai training. ByteDance's crawler. No operator documentation exists for it. Doubaobot — other bots. ByteDance's crawler for the Doubao assistant.
Confusing two of these is the most common mistake with this operator, and it is expensive in one direction: a rule meant for the training crawler that lands on the search crawler removes you from results without stopping any training.
One real request
Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; spider-feedback@bytedance.com)Questions
- Is Bytespider in my logs really ByteDance?
- There is no way to prove it. ByteDance publishes neither an address list nor a reverse DNS record for this crawler, so the user agent is the only evidence and anyone can send it. We record the visit as an unverified claim rather than presenting it as confirmed.
- How do I tell Bytespider apart from ByteDance's other crawlers?
- By the token, and only by the token. ByteDance also sends TikTokSpider, Doubaobot, and the strings differ by a few characters while the jobs differ completely — one may be answering somebody's question while another collects training material. Match on the exact token rather than on the operator's name appearing anywhere in the header, or the four end up counted as one.
- Does Bytespider run JavaScript?
- No. It requests the HTML and leaves. That is why a browser-based analytics tag never records these visits: the tag is JavaScript, and nothing runs it. Seeing this crawler at all requires reading it server-side.
- Why is Bytespider counted as Training and not something else?
- Because that is the job this particular crawler does. ByteDance runs more than one — TikTokSpider, Doubaobot — and they are counted separately because the decisions about them are separate. Filing them together under one operator name would let ordinary indexing look like AI interest, which is the single most misleading thing an AI traffic report can do.
- Why does Bytespider not show up in Google Analytics?
- Google Analytics runs in the browser. This crawler never opens a browser — it requests the HTML, reads it, and leaves, so the tracking script is never executed and no event is ever sent. Every browser-based analytics tool has the same blind spot, which is why crawler traffic has to be read from the server side to be seen at all.
- Why can a Bytespider visit not be verified?
- ByteDance publishes neither a machine-readable list of the addresses its crawler uses nor a reverse DNS record pointing back at itself. With neither, the user-agent header is the only evidence, and anyone can send it. Rather than showing a confidence we cannot support, the visit is recorded as an unverified claim — which is also the honest answer to give anyone asking how much of your AI traffic is real.
Related
See whether Bytespider is reaching your pages.
AI crawlers take your HTML and leave without running a line of JavaScript, so a browser tag records nothing. TrueStat reads them server-side, checks each address against what the operator publishes, and shows you which pages were fetched — including the ones that returned a 404.