The rise of the AI crawler
Vercel and MERJ find that AI crawlers now generate substantial web traffic but generally cannot render JavaScript and waste over one-third of requests on 404s. Sites that need AI visibility should server-render critical content and maintain disciplined URL hygiene.
The rise of the AI crawler
Author: Giacomo Zecchini, Alice Alexandra Moore, Malte Ubl, and Ryan Siddle | Published: 2024-12-17 | Generated: 2025-06-25 | Domain: vercel.com
Tags: ‘#ai’ ‘#web-crawlers’ ‘#javascript’ ‘#seo’ ‘#server-side-rendering’
TLDR
Vercel and MERJ analyzed AI-crawler traffic across
nextjs.org, Vercel’s network, and two job sites, finding nearly 1.3 billion monthly requests from GPTBot, Claude, AppleBot, and PerplexityBot—28% of Googlebot’s volume. Major AI crawlers such as GPTBot and ClaudeBot fetch JavaScript but do not execute it, making client-rendered content largely inaccessible; they also make inefficient requests, with more than 34% landing on 404s. Server-rendering critical content, keeping sitemaps current, and managing redirects are essential for visibility and crawler efficiency.
Key Takeaways
- AI crawler traffic is already significant: Googlebot made 4.5 billion monthly fetches across Gemini and Search, versus 569 million from GPTBot, 370 million from Claude, 314 million from AppleBot, and 24.4 million from PerplexityBot.
- Most major AI crawlers do not render JavaScript: OpenAI, Anthropic, Meta, ByteDance, Perplexity, and Common Crawl crawlers do not execute client-side JavaScript; Gemini inherits Googlebot’s rendering capabilities, while AppleBot uses a browser-based renderer.
- Fetching JavaScript does not make client-rendered content visible: ChatGPT directs 11.50% of requests to JavaScript and Claude 23.84%, but neither executes those files. Initial HTML, embedded JSON, and delayed React Server Components may still be readable.
- Crawler request patterns are inefficient: ChatGPT sends 34.82% of fetches to 404s and 14.36% through redirects; Claude hits 404s 34.16% of the time. Googlebot’s comparable rates are 8.22% for 404s and 1.49% for redirects.
- Site owners should optimize for accessibility: Server-render main content, metadata, and navigation; reserve client rendering for nonessential enhancements.
robots.txtand Vercel’s WAF can restrict AI crawlers when desired.
Images & Media
- AI crawler traffic comparison chart — Visualization comparing AI crawler request volume on Vercel’s network.
- AI crawler traffic comparison chart (dark theme) — Dark-theme version of the AI crawler traffic visualization.
Referenced Links
- How Google handles JavaScript throughout the indexing process — Vercel and MERJ’s related analysis of Googlebot rendering and indexing.
- MERJ — SEO consultancy that collaborated with Vercel on the crawler research.
- SSR, ISR, and SSG rendering strategies — Guidance for selecting rendering approaches that keep essential content crawlable.
- Applebot documentation — Apple’s crawler identification and access-control documentation.
- OpenAI crawler documentation — OpenAI bot user agents and crawler-control guidance.
- Block AI Bots Firewall Rule — Vercel WAF template for blocking recognized AI crawlers.