Cloudflare CEO Urges UK Regulator to Separate Google’s Search and AI Crawlers
Key Points
- Cloudflare CEO Matthew Prince is lobbying the UK CMA to regulate Google’s bundled search and AI crawlers.
- Prince argues that bundling gives Google an unfair advantage over other AI companies.
- Cloudflare provides data showing how Google’s crawler operates and why competitors can’t easily replicate it.
- 80% of AI companies are Cloudflare customers, positioning the firm as a neutral observer.
- Media publishers say blocking Google’s crawler also disables ad‑safety services, harming revenue.
- Industry leaders, including People, Inc. CEO Neil Vogel, echo concerns about Google’s practices.
- Prince proposes a market where AI firms pay for content rather than rely on free access.
- The CMA’s special status designation for Google could enable broader regulatory actions.
Cloudflare chief executive Matthew Prince is lobbying the U.K.’s Competition and Markets Authority to impose stricter rules on Google’s use of its web crawler for both search and AI products. Prince argues that bundling the crawler gives Google an unfair advantage over other AI companies, forcing media sites to opt out of both search and AI content simultaneously. Cloudflare, which hosts many AI firms, is providing data to the CMA to support regulatory action and is promoting a more competitive market where AI providers would pay for content rather than rely on Google’s free access.
Cloudflare’s Position on Google’s Crawler Bundling
Matthew Prince, chief executive of Cloudflare, traveled to London to discuss with the U.K.’s Competition and Markets Authority (CMA) the need for tighter regulation of Google’s web crawler. Prince says the CMA’s recent designation of Google as a “special status” player in search and advertising markets opens the door for broader oversight, including Google’s AI Overviews, AI Mode, Discover feed, Top Stories, and News tab.
Cloudflare is not an AI company itself, but it provides infrastructure to a large share of AI firms—Prince notes that “80% of the AI companies are our customers.” Because of that position, Cloudflare believes it can offer the regulator concrete data on how Google’s crawler operates and why it is difficult for competitors to replicate its reach.
The Problem With Bundled Crawlers
Prince contends that Google’s practice of using the same crawler for both its search engine and its AI products creates an “unfair advantage.” He explains that Google claims a “God‑given right” to access all web content without paying, leveraging the crawler it has built over “27 years.” This bundling forces website owners to opt out of both search and AI content at the same time, a scenario that is “not feasible for most,” especially media companies that rely on search traffic for a significant portion of revenue.
Blocking Google’s crawler also disables the company’s ad‑safety mechanisms, meaning that advertisements across platforms would stop working. Prince argues that this creates a “non‑starter” for publishers who might consider refusing Google’s crawler.
Calls for Competition and Payment for Content
Prince proposes that a competitive market would allow thousands of AI companies to purchase content from media outlets, rather than relying on a single free source. He praises the CMA’s move to flag Google as a potential regulatory target, describing it as a “thoughtful move” that acknowledges Google’s unique advantage.
Other industry leaders echo Prince’s concerns. Neil Vogel, CEO of People, Inc., the largest digital and print publisher in the United States, recently labeled Google a “bad actor” and highlighted that media firms feel compelled to let Google crawl their sites for AI content because of the bundled approach. Vogel’s company has adopted Cloudflare’s solution to block unpaid AI crawlers and reports that discussions are underway with several large language‑model providers.
Potential Outcomes
If the CMA adopts stricter rules, Google may be required to separate its search crawler from its AI data collection, leveling the playing field for other AI developers. Cloudflare’s data and advocacy aim to support that regulatory shift, encouraging an environment where AI firms must pay for the content they use, thereby fostering broader competition and protecting the interests of publishers.