# meta-webindexer > meta-webindexer is Meta's crawler for search indexing. Builds the index behind Meta AI search. Meta is indexing your page for its AI search results. **Category:** AI crawlers **Also known as:** Meta meta-webindexer **Updated:** 2026-09-09 **Source:** https://truestat.io/glossary/meta-webindexer --- ## What meta-webindexer is Meta's crawlers are among the highest-volume on the web — Cloudflare has reported meta-externalagent second only to Googlebot in request count. Meta publishes no address list, so none of its traffic can be verified, and its share of a site's bandwidth is often the surprise on this page. On this site meta-webindexer is counted as indexing, which is the row it appears in on your dashboard. The category matters because it decides which number moves when this crawler visits, and the four are reported separately for exactly that reason. ## What its traffic looks like A steady crawl over time rather than a burst: robots.txt first, then pages in waves, revisiting the ones that change often. Volume tracks how large and how frequently updated your site is. A sudden increase usually follows a sitemap change or a burst of new pages, and a sudden stop is worth investigating — it often means a robots rule or a server error rather than a decision by the operator. ## How it is verified Meta publishes neither a machine-readable list of the addresses this crawler uses nor a reverse DNS record pointing back at itself. The user-agent header is therefore the only evidence, and anyone can send it — so a visit is recorded as an unverified claim rather than as a confirmed fetch. That is not a gap in the measurement so much as an honest ceiling. A report that showed these as verified would be inventing a confidence nobody can support, and the number it produced would be the first thing to fall apart when somebody asked where it came from. ## If you block it You stop being findable through Meta. Whether that matters depends entirely on where your readers are; check your own numbers before deciding, rather than blocking by reflex. Treat this as a search-engine decision rather than an AI one, because that is what it is. The question is whether the people you want are looking through this engine, and your own analytics answer it better than any general advice can. It is worth checking what you actually lose before writing the rule. Blocking an index crawler is one of the few changes here with an effect that compounds quietly: nothing breaks on the day, and traffic simply stops arriving over the following weeks. ## How it differs from Meta's other crawlers Meta sends 4 crawlers and gives each its own token, so they can be allowed and refused separately. The tokens differ by a few characters and what the fetch means does not: meta-externalfetcher — ai assistant. Fetches a specific URL a person asked Meta AI about. meta-externalagent — ai training. Meta's training corpus crawler. meta-externalads — other bots. Meta's crawler for advertising and business products. Confusing two of these is the most common mistake with this operator, and it is expensive in one direction: a rule meant for the training crawler that lands on the search crawler removes you from results without stopping any training. ## Questions ### Is meta-webindexer in my logs really Meta? There is no way to prove it. Meta publishes neither an address list nor a reverse DNS record for this crawler, so the user agent is the only evidence and anyone can send it. We record the visit as an unverified claim rather than presenting it as confirmed. ### How do I tell meta-webindexer apart from Meta's other crawlers? By the token, and only by the token. Meta also sends meta-externalfetcher, meta-externalagent, meta-externalads, and the strings differ by a few characters while the jobs differ completely — one may be answering somebody's question while another collects training material. Match on the exact token rather than on the operator's name appearing anywhere in the header, or the four end up counted as one. ### Does meta-webindexer run JavaScript? No. It requests the HTML and leaves. That is why a browser-based analytics tag never records these visits: the tag is JavaScript, and nothing runs it. Seeing this crawler at all requires reading it server-side. ### Why is meta-webindexer counted as Indexing and not something else? Because that is the job this particular crawler does. Meta runs more than one — meta-externalfetcher, meta-externalagent, meta-externalads — and they are counted separately because the decisions about them are separate. Filing them together under one operator name would let ordinary indexing look like AI interest, which is the single most misleading thing an AI traffic report can do. ### Why does meta-webindexer not show up in Google Analytics? Google Analytics runs in the browser. This crawler never opens a browser — it requests the HTML, reads it, and leaves, so the tracking script is never executed and no event is ever sent. Every browser-based analytics tool has the same blind spot, which is why crawler traffic has to be read from the server side to be seen at all. ### Why can a meta-webindexer visit not be verified? Meta publishes neither a machine-readable list of the addresses its crawler uses nor a reverse DNS record pointing back at itself. With neither, the user-agent header is the only evidence, and anyone can send it. Rather than showing a confidence we cannot support, the visit is recorded as an unverified claim — which is also the honest answer to give anyone asking how much of your AI traffic is real. ## Related - https://truestat.io/glossary/meta-externalfetcher - https://truestat.io/glossary/meta-externalagent - https://truestat.io/glossary/meta-externalads - https://truestat.io/glossary/fcrdns - https://truestat.io/glossary/spoofed-crawler --- From the TrueStat glossary — https://truestat.io/glossary. Privacy-first web analytics that also shows you which AI assistants are reading your site.