Special thanks to Rohin Lohe at Cloudflare, Jennifer Ong and Kyle Corcoran at TollBit, Tim Conard and Cuy Sheffield at Visa, and Son Do at Parallel, for feedback and contributions.
On March 19th, 2026, Matthew Prince, the CEO of Cloudflare, the world’s largest content delivery network sitting at ~20% of the web, predicted that bot traffic, driven by the rise of generative AI, would exceed human traffic in 2027. Just a few months later, Prince announced that agentic traffic had grown so quickly that bots had already overtaken human traffic. Bots account for roughly 60% of HTTP requests for HTML content across Cloudflare’s network, compared with 38.2% from humans. As automated systems generate a larger share of HTML page requests than humans, and AI agents account for a growing share of that machine traffic, the underlying business model of the internet is breaking.
The world wide web has enabled unprecedented global democratization of information, a mission which has been funded by a straightforward content-for-traffic barter exchange, wherein publishers would publish content on the web, and search engines would direct readers to their sites if it matched what they were looking for. Publishers provided information or entertainment to the web in the form of content; search engines helped direct readers to relevant webpages; and advertisers would monetize the user visit by displaying targeted ads on the webpages users were led to. Together, this synchronized loop established the multi-billion dollar advertising machine that funded the open, paywall-free web.
The exchange funded a vast amount of freely accessible information while concentrating distribution power in search and advertising platforms. It rewarded behavioral advertising, SEO-optimized volume, and engagement-driven content, leaving publishers dependent on referral and attention-based traffic. AI is chipping away at the referral leg from an already fragile bargain before a replacement was ready.
Google's advertising revenue grew from $66.9 million in 2001, its first fully disclosed advertising year, to $294.7 billion in 2025, a factor of about 4,400.
The explosive growth of AI chatbot applications in recent years, however, has introduced a new paradigm for discovering information on the web. By design, these AI apps prioritize user convenience by keeping the audience within the chat interface itself. Though convenient for users, this feature has become particularly detrimental for publishers. While search engines directed readers to websites where the content they’re looking for originates from, AI applications instead condense and summarize the information and present it as part of their final response to a given user prompt. It becomes harder for publishers to monetize web traffic as they lose control of the direct user relationship and start to receive less human web traffic.
One emerging solution is to introduce new "pay-per-request" payments infrastructure for content and search on the internet. In this paper, we explore how AI is “breaking the internet”, the two distinct approaches to this solution, and what the state of the newly emerging pay-per-request web looks like today.
A search engine uses a crawler, an automated program designed to index webpages, to build a ‘map’ of the web. Under the traditional content-for-traffic model, publishers allowed these crawlers to copy their text for free because the search engine rewarded them by routing human readers back to their websites. To manage this access, the industry relied on a text file called robots.txt, an unencrypted, voluntary honor system. Robots.txt worked smoothly for as long as it did because defying robots.txt meant losing access to publishers’ content, which negatively impacts the search engine's index of the web.
Generative AI applications disrupt the web’s content-for-traffic barter exchange by turning crawlers from indexers into active consumers, which unfolds most evidently in training and real-time retrieval. During training, a large language model like GPT or Claude learns from a massive corpus of data scraped from the open web, which includes various books, articles, code, and art. The model trains on the totality of this human knowledge and serves it back to users with no link to the source and no payment to the creator. When retrieving information on the web in real-time to answer a user prompt, the model indexes the live web at the exact moment a user asks a question, pulls the relevant passage, and reformats it. Rather than handing the reader the original page, the model restates the information into a fresh, inline reply that answers the prompt entirely. A crawler indexes a page once, and the model can serve what it found to millions of readers, none of them ever carrying the reader back to the source.
When taking the mechanics of extraction, alongside behavioral data, federal legal battles, and artistic exploitation, it becomes evident that the open web's foundation is experiencing a structural collapse, as the impact is already showing up across several interconnected domains.
The behavioral impact of AI crawlers can be measured by the ‘crawl-to-refer’ ratio: the number of pages a crawler scrapes versus the number of actual users it sends back. Data from Cloudflare shows that in the past six months alone, both training (43.9%) and training and search (41.5%) drove over 85% of identified AI crawler requests observed across Cloudflare’s network.Googlebot still runs nearly five pages for every visitor it sends back, 5.3 over the past year, which is more in line with historical exchanges, and every major AI platform sits one to three orders of magnitude above it. Over the past year, Anthropic’s ClaudeBot averaged 4.2k page crawls for every single visitor sent back to an origin site, Mistral 884, OpenAI 731, and Perplexity 147. Cloudflare does note that referrals from native apps don’t carry referral headers, so these ratios could overstate the imbalance. A Pew Research Center study found that when AI summaries intercept search queries, human click-through rates drop from 15% to 8%, direct click attribution with summaries drops to 1%, and users abandon browsing sessions entirely 26% of the time.
AI crawlers have also prompted legal action. In Bartz v. Anthropic, Judge William Alsup ruled that training on lawfully acquired books was fair use because the use was transformative, meaning it created something new rather than substituting for the books themselves, while the claim over the pirated libraries Anthropic had downloaded moved toward trial. Anthropic settled that claim privately for $1.5 billion, roughly $3,000 per work across some 500,000 titles. The outcome was widely read as a win for Anthropic, because the fair-use ruling on training stood and the settlement closed off statutory damages that could have run far higher. As the case settled, no court ever established what training on pirated content actually costs. Meanwhile the New York Times v. OpenAI and Microsoft case is still ongoing, forcing OpenAI to turn over 20 million ChatGPT conversation logs as part of an investigation into how closely models reproduce articles to substitute for paid subscriptions.
These cases highlight that the costs of the unmetered extraction of AI crawlers falls on independent creators and visual artists in particular. When OpenAI released its Studio Ghibli-style rendering capabilities in early 2025, it highlighted just how thin existing copyright protections are. Because U.S. law protects specific expressions of a work rather than its overall style, AI models can systematically clone an artistic aesthetic built over decades of human labor without naming or compensating the creators. OpenAI's policy blocks prompts for a living artist like Hayao Miyazaki, yet completely permits prompts for "Studio Ghibli”. A specific character like Totoro is protected, but the general Ghibli style is not, which leaves the artist with little legal ground even though the model could produce that look only because it trained on the studio's films. Pay-per-request infrastructure cannot resolve every harm described above. It can meter and compensate for future access to a work, but it cannot retroactively compensate artists for material already used in training or prevent a model from reproducing a style it has already learned. Those harms require separate licensing, copyright, and enforcement remedies.