# When the ten blue links disappear > On 15 September Cloudflare changes who may read a website automatically. Why that turns into a business decision, and what comes after search engine optimisation. - **Author:** Dominic Asche - **Date:** 09 / 2026 - **Reading time:** 6 min - **URL:** https://fse-group.de/en/think-tank/when-the-ten-blue-links-disappear The bargain of the web read: crawl the content, appear in search, get visitors, monetise visitors. AI moves the last link of that chain, and from 15 September Cloudflare turns it into a decision every operator has to make. On 15 September 2026 Cloudflare changes the defaults for automated access to websites. That sounds like a configuration question. It touches the bargain the web has been built on for twenty years: the trade between search engines and the people who publish content. For a long time the deal read: crawl the content, appear in search, get visitors, monetise visitors. Every link in that chain had its price, and at the end of it a person stood on a page. AI moves the last link. Crawl the content, process the information, serve the answer directly, possibly send no visitor at all. For publishers that difference matters: their content can still be the basis of an answer while the traffic disappears that advertising and subscriptions were sold against. ![Two-part cartoon. The left panel is headed The Old Web, subtitled Search brings people to your website, with the line: a simple deal, search engines get content, websites get visitors. Four steps run across it: a Google-branded robot crawls content, the page appears in Google search for the query best hiking trails, a person with a backpack walks up to the site, and the site shows a Book now button with cart, card and mail icons under the note that visitors become customers. A box at the foot reads: good content brings people, people create value. The right panel is headed The New Web, subtitled AI search delivers the answer, with the line: content is still read and used, but often without sending people to your website. Its four steps: an AI robot crawls the same content, a second robot processes it by summarising, understanding, comparing and generating an answer, an AI search then lists three Alpine trails directly, and possibly no visitor is sent at all. The person sits in an armchair with a mug, thinking that this is exactly the information they needed. The box at the foot reads: the content remains valuable, but the click is no longer guaranteed.](https://fse-group.de/media/think-pieces/20260907_thinktank-blue-links-welten-en.png) ### A business decision, not a setting Cloudflare will sort automated traffic not just by sender but by purpose: search, agent and training. That looks like a clean split. What happens to crawlers serving several purposes at once is the more interesting part. Cloudflare counts Googlebot, Applebot and BingBot among these multi-purpose crawlers. For them the most restrictive rule configured will apply. So blocking training may also block a crawler that serves search at the same time. [1] A technical setting thus turns into a trade-off with a price attached: do I want to stop my content being used for certain AI purposes if that costs me visibility in search? Matthew Prince, CEO of Cloudflare, puts it more bluntly. On the Big Technology Podcast on 2 September he announced that Google would be blocked by default for advertising- or subscription-funded offerings unless the operator objects. [2] At Google the line runs elsewhere. Google-Extended has for some time let operators control whether crawled content may be used to train future Gemini models and for certain grounding applications. Google states that this setting affects neither inclusion in search nor ranking. [3] AI Overviews and AI Mode inside Google Search, however, count as part of search for Google. Anyone who does not want them has to block Googlebot, and loses search along with them. The choice is no longer whether a machine may read. It is for what purpose, at what price, and in return for what. ### When the answer replaces the click That AI search produces less website traffic is by now more than a hunch. Pew Research studied the browsing behaviour of 900 US adults and analysed 68,879 Google searches. When an AI summary appeared, users clicked a classic search result in 8 percent of cases. Without a summary it was 15 percent. They opened a source link named inside the summary on 1 percent of those visits. [4] An experimental study published in 2026 randomly assigned users to a search with or without AI Overviews. Where an overview appeared, organic outbound clicks fell by 39.8 percent and searches without any click rose by 34.5 percent. [5] The content can stay valuable throughout. What drops is its traffic value. ![Infographic titled When the answer replaces the click. At the top a chain of four steps: content gets crawled, the AI processes it, the answer is served directly, the visitor never arrives. Below a bar comparison titled An AI summary halves the click rate, measured by Pew Research across 900 US adults and 68,879 Google searches: without a summary 15 percent of searches end in a click on a search result, with a summary 8 percent, and only 1 percent leads to a source cited inside the summary. An experiment by Agarwal and Sen from 2026 confirms the direction: organic clicks fall by 39.8 percent and searches without any click rise by 34.5 percent. At the foot three conclusions: the content stays valuable while its traffic value drops; the website visit is no longer the only unit of distribution; search engine optimisation turns into an access strategy.](https://fse-group.de/media/think-pieces/20260907_thinktank-blue-links-klickrate-en.png) ### What does a crawl cost? Cloudflare’s answer is not only about shutting bots out. With Pay Per Crawl, participating operators can allow an AI crawler, block it, or require payment. Cloudflare currently names 0.001 US dollars per successful crawl as the minimum price. [6] The amount alone would hardly be news. The mechanism is what matters: a crawler can state the highest price it will accept. If the offer does not match, the server answers with status code 402, Payment Required. [7] Access itself becomes a transaction: request, price, payment, access. For the first time machine access can be tied to an economic condition rather than only technically allowed or denied. ### Publishers and retailers do different arithmetic For a publisher, articles, archives and research can be the actual product. Wanting to license their use by AI providers makes sense, and that market already exists. OpenAI has announced agreements with Axel Springer, the Financial Times, The Atlantic, Condé Nast and the Washington Post. [8] For an online shop the arithmetic runs differently. A product description is not meant to monetise a page view, it is meant to sell a product. An agent reading price, properties, availability and delivery terms can be valuable to the retailer even though no human ever saw the product page. Charging that agent for the crawl would be self-defeating. Content businesses want to monetise the access. Commerce businesses want to monetise the outcome. For retailers the more interesting question is therefore a different one: how do they make sure their products are discoverable, comparable and purchasable for buying agents? Because once an agent picks on the customer’s behalf, optimisation shifts from the human page visit to the quality of the underlying data. An established company starts from a different place than a new entrant: reviews, product information, mentions and delivery histories already exist. A buying agent needs other things than a person, though: structured product data, unambiguous prices, current availability, reliable company data and ideally a way to carry out the transaction directly. For small suppliers that can turn into a distribution problem. It can just as easily be an opening, because whoever gets compared objectively needs less name recognition. ### What follows SEO is not GEO, it is an access strategy Companies have built their websites for two audiences so far: people, and the search engines that send people. Now a third one arrives that reads, compares, decides and increasingly acts by itself. The split between public and not public no longer covers that. An access strategy needs four answers: - **Discover:** information that should be found, indexed and cited as widely as possible. - **Transact:** product, price and service information that agents can process reliably and use for transactions. - **Monetize:** content, data, interfaces or services whose machine access carries value in itself. - **Protect:** content and uses for which automated access is not wanted. Not every company needs all four. But every company with meaningful digital distribution will have to work through this trade-off, and preferably before a default setting makes the call for them. Search engine optimisation is not finished by any of this. Anyone who wants to appear as a source in AI answers has to be in the index. But the website click is no longer the only relevant outcome of being findable. A company can turn up in an answer, be considered in a purchase decision or supply data to an agent without any of it ever becoming a session in the analytics tool. So the decisive change may not be that fewer people click ten blue links. It is that the website visit is no longer necessarily the unit in which digital distribution is measured and paid for. #### Sources - [1] Cloudflare, Jin-Hee Lee and Bryan Becker: [*Your site, your rules: new AI traffic options for all customers*](https://blog.cloudflare.com/), 1 July 2026. New traffic classification, defaults from 15 September 2026 and the handling of multi-purpose crawlers. - [2] Big Technology Podcast, Alex Kantrowitz with Matthew Prince: *Cloudflare CEO: We’re Ready To Block Millions Of Websites From AI*, 2 September 2026. - [3] Google for Developers: [*Google’s common crawlers, Google-Extended*](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers), last updated 14 July 2026. - [4] Pew Research Center, Athena Chapekis and Anna Lieb: [*Google users are less likely to click on links when an AI summary appears in the results*](https://www.pewresearch.org/short-reads/2025/07/22/google-users-are-less-likely-to-click-on-links-when-an-ai-summary-appears-in-the-results/), 22 July 2025. - [5] Saharsh Agarwal and Ananya Sen: [*The Impact of Google AI Overviews on Publisher Traffic and User Experience*](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5174691), SSRN, 3 April 2026, revised 8 July 2026. - [6] Cloudflare Developers: [*Pay Per Crawl*](https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/). - [7] Cloudflare, Rohin Lohe, Justin Ridgely and Will Papper: [*Announcing the Monetization Gateway: charge for any resource behind Cloudflare via x402*](https://blog.cloudflare.com/), 1 July 2026. - [8] OpenAI: [*We’re bringing the Financial Times’ world-class journalism to ChatGPT*](https://openai.com/index/content-partnership-with-financial-times/), 29 April 2024, plus the published partnerships with Axel Springer, The Atlantic, Condé Nast and the Washington Post.