Cloudflare revealed its sixth annual Yr in Evaluate, providing a complete appears to be like at Web visitors, safety, and AI crawler exercise throughout 2025.
The report attracts on information from Cloudflare’s community, which spans greater than 330 cities throughout 125 nations and handles over 81 million HTTP requests per second on common.
The AI crawler findings stand out. Googlebot crawled much more internet pages than another AI bot, reflecting Google’s dual-purpose method to crawling for each search indexing and AI coaching.
Googlebot Prime AI Crawler Site visitors
Cloudflare analyzed profitable requests for HTML content material from main AI crawlers throughout October and November 2025. The outcomes confirmed Googlebot reached 11.6% of distinctive internet pages within the pattern.
That’s greater than 3 occasions the pages seen by OpenAI’s GPTBot at 3.6%. It’s practically 200 occasions greater than PerplexityBot, which crawled simply 0.06% of pages.
Bingbot got here in third at 2.6%, adopted by Meta-ExternalAgent and ClaudeBot at 2.4% every.
The report famous that as a result of Googlebot crawls for each search indexing and AI mannequin coaching, internet publishers face a tough selection. Blocking Googlebot’s AI coaching means risking search discoverability.
Cloudflare wrote:
“As a result of Googlebot is used to crawl content material for each search indexing and AI mannequin coaching, and due to Google’s long-established dominance in search, Site operators are basically unable to dam Googlebot’s AI coaching with out risking search discoverability.”
Associated: Full Crawler Checklist For AI Person-Brokers
AI Bots Now Account For 4.2% of HTML Requests
All through 2025, AI bots (excluding Googlebot) averaged 4.2% of HTML requests throughout Cloudflare’s buyer base. The share fluctuated between 2.4% in early April and 6.4% in late June.
Googlebot alone accounted for 4.5% of HTML requests, barely greater than all different AI bots mixed.
The share of human-generated HTML visitors began 2025 at seven share factors beneath non-AI bot visitors. By September, human visitors started exceeding non-AI bot visitors on some days. As of December 2, people generated 47% of HTML requests whereas non-AI bots generated 44%.
Crawl-to-Refer Ratios Present Huge Variation
Cloudflare tracks how typically AI and search platforms ship visitors to websites relative to how typically they crawl. A excessive ratio means heavy crawling with out sending customers again to supply websites.
Anthropic had the best ratios amongst AI platforms, starting from roughly 25,000:1 to 100,000:1 throughout the second half of the 12 months after stabilizing from earlier volatility.
OpenAI’s ratios reached as excessive as 3,700:1 in March. Perplexity maintained the bottom ratios amongst main AI platforms, typically beneath 400:1 and underneath 200:1 from September onward.
For comparability, Google’s search crawl-to-refer ratio stayed a lot decrease, typically between 3:1 and 30:1 all year long.
Person-Motion Crawling Grew Over 20X
Not all AI crawling is for mannequin coaching. “Person motion” crawling happens when bots go to websites in response to consumer questions posed to chatbots.
This class noticed the quickest development in 2025. Person-action crawling quantity elevated greater than 15 occasions from January by means of early December. The pattern intently matched the visitors sample for OpenAI’s ChatGPT-Person bot, which visits pages when customers ask ChatGPT questions.
The expansion confirmed a weekly utilization sample beginning in mid-February, suggesting elevated use in colleges and workplaces. Exercise dropped throughout June by means of August when college students had been on break and professionals took holidays.
AI Crawlers Most Blocked In Robots.txt
Cloudflare analyzed robots.txt information throughout practically 3,900 of the highest 10,000 domains. AI crawlers had been probably the most regularly blocked consumer brokers.
GPTBot, ClaudeBot, and CCBot had the best variety of full disallow directives. These directives inform crawlers to keep away from total websites.
Googlebot and Bingbot confirmed a special sample. Their disallow directives leaned closely towards partial blocks, doubtless targeted on login endpoints and non-content areas relatively than full website blocking.
Civil Society Grew to become Most-Attacked Sector
For the primary time, organizations within the “Folks and Society” vertical had been probably the most focused by assaults. This class contains non secular establishments, nonprofits, civic organizations, and libraries.
The sector obtained 4.4% of world mitigated visitors, up from underneath 2% in the beginning of the 12 months. Assault share jumped to over 17% in late March and peaked at 23.2% in early July.
Many of those organizations are protected by Cloudflare’s Mission Galileo.
Playing and video games, the most-attacked vertical in 2024, noticed its share drop by greater than half to 2.6%.
Different Key Findings
Cloudflare’s report included a number of further findings throughout visitors, safety, and connectivity.
International Web visitors grew 19% year-over-year. Development stayed comparatively flat by means of mid-April, then accelerated after mid-August.
Submit-quantum encryption now secures 52% of human visitors to Cloudflare, practically double the 29% share in the beginning of the 12 months.
ChatGPT remained the highest generative AI service globally. Google Gemini, Windsurf AI, Grok/xAI, and DeepSeek had been new entrants to the highest 10.
Starlink visitors doubled in 2025, with service launching in additional than 20 new nations.
Practically half of the 174 main Web outages noticed globally had been brought on by government-directed shutdowns. Cable lower outages dropped practically 50%, whereas energy failure outages doubled.
European nations dominated Web high quality metrics. Spain topped the record for general Web high quality, with common obtain speeds above 300 Mbps.
Why This Issues
The AI crawler information ought to impacts how you concentrate on bot entry and visitors.
Google’s dual-purpose crawler creates a aggressive benefit. You possibly can block different AI crawlers whereas retaining Googlebot entry for search visibility, however you possibly can’t separate Google’s search crawling from its AI coaching crawling.
The crawl-to-refer ratios assist quantify what publishers already suspected. AI platforms crawl closely however ship little visitors again. The hole between crawling and referring varies broadly by platform.
The civil society assault information issues should you work with nonprofits or advocacy organizations. These teams now face the best fee of assaults.
Wanting Forward
Cloudflare expects AI metrics to vary because the area continues to evolve. The corporate added a number of new AI-related datasets to this 12 months’s report that weren’t accessible in earlier editions.
The crawl-to-refer ratios could change as AI platforms regulate their search options and referral conduct. OpenAI’s ratios already confirmed some decline by means of the 12 months as ChatGPT search utilization grew.
For robots.txt administration, the info exhibits most publishers are selecting partial blocks for main search crawlers whereas absolutely blocking AI-only crawlers. The year-end state of those directives gives a baseline for monitoring how writer insurance policies evolve in 2026.
Featured Picture: Mamun_Sheikh/Shutterstock
