Shoal Research Report: Bots Account for 60% of HTML Requests

Huo Xing Cai Jing reports that research firm Shoal Research found robots account for approximately 60% of HTTP requests for HTML content on Cloudflare’s network, with humans making up 38.2%. Cloudflare CEO Matthew Prince previously predicted that robot traffic would surpass human traffic by 2027, and has since announced that agent-driven traffic has already caused robots to exceed human traffic. The report states that AI applications keep users within chat interfaces and summarize content rather than directing them back to original sites. AI crawlers exhibit significantly higher rates of scraping and revisiting compared to Googlebot; over the past year, ClaudeBot averaged approximately 4,200 pages scraped per revisit. Pew Research shows AI summaries have reduced click-through rates from 15% to 8%. The traditional models of robots.txt and content-for-traffic are no longer sustainable. Litigation and private licensing serve only as temporary measures. Pay-per-request models are divided into closed markets with keys and open protocols using HTTP 402 and stablecoins, such as x402 and MPP. Most websites still offer content for free or block access entirely.