Meta's AI crawler sent 9.1 billion requests in Q2, triggering server alerts.

According to monitoring by Beating, Meta is aggressively scraping web pages to index its own AI search engine. Independent developer Pieter Levels reported that Meta frequently scraped multiple websites under his ownership this week, causing load alerts to trigger multiple times on one of his VPS servers. DataDome statistics show that in the second quarter of this year, Meta’s two AI crawlers issued a total of 9.1 billion requests. The Meta-WebIndexer, responsible for building search indexes, saw a 163% quarterly increase, with its request volume surpassing that of the training crawler for the first time in June. Meta has been known since 2024 to be developing its own search engine to reduce reliance on Google and Bing for Meta AI. Now, this search indexing system is rapidly expanding—but such intensive scraping brings virtually no real users back to websites. A Meta employee privately revealed that the company aims to build its own “Google” to avoid sending AI search requests to Google, only for Google to use them for training. This information remains unconfirmed by Meta and is currently based solely on anonymous sources.