Cloudflare says its OpenAI pilot added hundreds of millions of pages to ChatGPT search

What Cloudflare and OpenAI announced

Cloudflare and OpenAI announced a research pilot on 8 July 2026 to test whether data from Cloudflare's network could help AI search engines find web content faster. On 15 October they published the first results in a post on the Cloudflare blog credited to Cloudflare's Matthew Conroy and Jack Galilee alongside OpenAI's Henry Aspegren and James Balfour. Their headline claim is that the pilot “has enabled OpenAI to discover and index hundreds of millions of new web pages” that ChatGPT can now draw on.

The pilot did not run across all of Cloudflare's customers. The post says it used “a participating cohort of ~1% of Cloudflare's total network.” That is a small slice of a large footprint: Cloudflare's July press release says more than 20% of the web sits behind its network. Neither company named the participating sites or said how many domains the 1% represents.

The July announcement framed the project around accuracy and freshness. “Up-to-date information is important for delivering accurate answers to people using ChatGPT,” OpenAI's VP of Research Nick Ryder said in the Cloudflare press release. Cloudflare CEO Matthew Prince said sharing network signals could “make AI search more efficient and help people get quality answers faster.” The October post calls the early answer to that question “a resounding yes.”

How the discovery signals work

Cloudflare sits in front of a large share of the web as a content delivery network and security proxy, so it sees requests and responses for pages that may have few or no inbound links. Traditional crawlers mostly find pages by following links, which means a well-written manual, a current price list or a new menu can stay invisible for a long time. The post argues that these specific pages are exactly what people now ask AI systems about.

In the pilot, Cloudflare passes signals to OpenAI's search crawler about which addresses exist and are worth visiting. The July release listed the kinds of insight involved as “content freshness, traffic quality, and actual page changes.” OpenAI's crawler then fetches the page, and “OpenAI then independently evaluates and ranks how those pages appear based on how relevant and helpful they are,” the post says. The division matters: Cloudflare suggests where to look, but it does not decide what ChatGPT cites.

The change signals are the efficiency argument. A crawler that keeps re-downloading an unchanged page adds load for the site owner without benefit. If Cloudflare can tell the crawler when a page has actually changed, it can revisit the pages that matter and skip the rest. The authors are candid that the net effect is not always lower load: “Finding new pages can initially increase crawling as those pages are evaluated and indexed.”

The post ties the work to OpenAI's broader push to make ChatGPT a place to shop, learn and find answers, citing GPT-6 and Intelligent UI in ChatGPT. Better discovery is meant to feed those answers with more current sources, and to link back to them.

Filtering junk and flagging harmful pages

A large part of the post is about what not to crawl. The authors say they learned “how easily automated traffic can make a nonexistent page look interesting.” Bots repeatedly request files that do not exist, some sites redirect almost any address to their homepage, and other addresses lead to copies of the same content. A naive system watching traffic would treat all of these as new pages worth indexing.

Recognising those patterns lets Cloudflare filter addresses before OpenAI's crawler spends a request on them. The post describes this as cutting wasted work “in a way that benefits site owners,” since every avoided fetch is load the origin server does not have to serve.

The pilot is also testing signals that flag potentially harmful pages. The example given is a legitimate site with a single compromised page, such as a fake login form planted by an attacker. Flagging that page alone, rather than the whole domain, would keep it out of an index without penalising the site owner for the rest of the site. The authors say only that “early results suggest” these signals help; they give no detection rates or false-positive figures.

What site owners control

Both companies stress that the signals are not a permission slip. “A signal about a page doesn't grant permission to access it,” the post says, and the research “uses metadata about the web for search and discovery, rather than sharing page content for model training.” In practice that means OpenAI's crawler still has to fetch the page itself, and the site's normal access rules still apply when it does.

On the OpenAI side, those rules are set with robots.txt. OpenAI's crawler documentation separates OAI-SearchBot, which surfaces sites in ChatGPT search, from GPTBot, which gathers content that may be used to train its foundation models. The two settings are independent, so a site can allow search and block training. Sites that opt out of OAI-SearchBot will not appear in ChatGPT search answers, though they can still show up as navigational links, and OpenAI says robots.txt changes take about 24 hours to take effect.

On the Cloudflare side, the post points owners to AI Crawl Control, formerly called AI Audit and available on all plans. It shows which AI crawlers visit a site, lets owners set allow or block rules per crawler, tracks which crawlers obey robots.txt, and offers pay-per-crawl pricing in private beta. The pilot post does not say whether sites in the 1% cohort opted in individually or were selected by Cloudflare under existing terms.

Why this matters for AI search

For publishers and small sites, discovery has become a second distribution problem. Ranking on a search results page was the old goal; now a page also has to be found, fetched and judged useful by AI systems that answer questions directly. Cloudflare's pitch is that its network view can surface pages that link-based crawling misses, and that the same view can reduce the crawl load that AI bots place on origin servers.

For OpenAI, the pilot is a way to compete on freshness without crawling the whole web more often, which the post notes “would be expensive for everyone.” It also fits a pattern of OpenAI building commercial plumbing around ChatGPT, from a separate crawler that checks ad landing pages as ChatGPT tests visual ads to retail integrations. For Cloudflare, which sells tools to block AI crawlers or charge them per crawl, the pilot shows the other side of its position: helping a crawler it approves of work better.

The arrangement raises an obvious competition question that the post does not address: whether other AI search providers will get the same signals. It is described as a joint pilot with OpenAI, and nothing in either announcement says the data will be offered to other companies.

What the results do not show yet

The evidence published so far is a single aggregate figure, “hundreds of millions” of newly indexed pages, with no breakdown by site type, language or quality. Indexing a page is not the same as citing it, and neither company reports how often the new pages appear in ChatGPT answers, how much referral traffic they send back, or whether answers became more accurate. The post says measuring “what happens after content is discovered” and “real business outcomes” is the job of the next phase.

The claims are the companies' own and have not been independently checked. AI Lookout could not verify the page count, the cohort size or the harmful-page results. Readers running sites behind Cloudflare should watch for the next update, which promises better page selection and faster change signals, and check their robots.txt and AI Crawl Control settings so that what OpenAI's crawlers can do matches what they intend.