Báo cáo nghiên cứu Shoal: Bot chiếm 60% số yêu cầu HTML

The Huoxing Finance report states that research firm Shoal Research found robots account for approximately 60% of HTTP requests for HTML content on the Cloudflare network, while humans make up 38.2%. Cloudflare CEO Matthew Prince previously predicted that robot traffic would surpass human traffic by 2027, and later announced that the growth in agent traffic has already caused robots to exceed human traffic. The report notes that AI applications keep users within chat interfaces and summarize content rather than driving them back to original sites. AI crawlers have significantly higher rates of scraping and revisiting compared to Googlebot; over the past year, ClaudeBot averaged approximately 4,200 pages scraped per revisit. Pew Research shows that AI summaries have reduced click-through rates from 15% to 8%. The robots.txt model and content-for-traffic paradigm are becoming unsustainable. Litigation and private licensing are temporary solutions. Pay-per-request models are divided into closed markets with keys and open protocols using HTTP 402 and stablecoins, such as x402 and MPP. Most websites still offer content for free or block access.