When Answers Replace Links

Much of the commercial web has been running on a deal that nobody signed.

A publisher lets a search engine send a program, a crawler, to copy its pages. The search engine builds an index and shows people a ranked list of links. Someone clicks, lands on the publisher's site, sees an ad or subscribes, and that money pays for the next page. The publisher's half of the deal is permission. A small file called robots.txt, sitting at the root of every site, tells each crawler by name what it may fetch, and publishers left the door open because crawling led to visits.

Now look at what happens when Google puts an AI-written summary above the results. In browsing data from March 2025, Pew Research Center found that people clicked a regular result link in 8 percent of visits to a results page with a summary, against 15 percent without one. They clicked a link inside the summary itself in 1 percent. And the visit was more likely to end the browsing session entirely.

That's the whole story in one number: the crawl still happens, and the visit that used to pay for it becomes optional. An AI answer reads the same pages a search engine always did and keeps the visit for itself. Value moves away from pages that answer common questions, toward sources an answer can't replace and toward whoever owns the answer. Blocking crawlers, licensing content, suing and optimizing each address a different piece of that, and none restores the old deal on its own.

How an answer breaks the exchange

An answer engine is a search product that writes the response itself instead of handing you a list of links. Google's version in its search results is called AI Overviews. ChatGPT, Perplexity and Claude each offer a search mode that answers with a few citations.

Mechanically, the first steps look the same as old search. The service fetches pages, or relies on what its model absorbed from them during training. Then, instead of ranking links, a language model reads the most relevant pages and writes a few paragraphs that answer the question, usually with a handful of small source links attached.

Scroll sideways to see the whole figure

The crawl is the same size or larger. The reader has much less reason to open the source, because the useful part of the source is already on the screen. Labs want to own that answer surface for exactly this reason: whoever writes the answer holds the reader's attention (Why Labs Build Products).

There's a simple measure for how badly the exchange has broken. The crawl-to-referral ratio counts how many pages a service fetches from websites for each visit it sends back. Cloudflare, which sits in front of a large share of the world's websites, published these numbers through 2025. Google's ratio moved between about 3 and 30 pages per visit. Perplexity's mostly stayed below 400. OpenAI's spiked into the thousands. Anthropic's ran in the tens of thousands for most of the year.

Some of that crawling is for training models, which was never meant to send anyone back, and native apps don't always identify the visits they send, so some ratios are overstated. But the direction is clear. A ratio of thousands means a service takes pages and returns almost nothing a publisher can sell.

What the clicks show, and what Google says

Pew's study is the best documented, and it's worth reading carefully. The summaries appear mostly on longer, question-shaped searches: on 8 percent of one- or two-word searches, and on more than half of searches ten words or longer. Those searches may have different click habits whatever the page shows. A fair reading is that people click about half as often when a summary is present, with part of that gap explained by which searches get summaries.

Other data points the same way. Chartbeat, which measures traffic for news sites, found Google search traffic to 2,576 publisher sites down a third globally over the year to November 2025, though how much of that is due to AI summaries is unclear. Search-marketing firms report bigger click drops on queries that carry a summary, after subtracting the general decline.

Google disputes the picture. In August 2025 the head of Google Search wrote that total clicks from search were relatively stable and that click quality had improved slightly, with traffic shifting toward forums, video, podcasts and first-hand content. The post gave no numbers or method.

Both can be true at once. A total can hold steady while its distribution changes, and a reference site can lose half its traffic while a forum gains. Meanwhile Alphabet reported Google Search revenue of $63.3 billion in the second quarter of 2026, up about 17 percent from a year earlier, with its chief executive crediting AI features for growth in queries.

Total clicks and clicks to your site are different measurements, and only the second one pays your bills.

Who is exposed

Exposure comes down to one question: would the reader stop at the answer?

Reference pages, how-to guides, definitions, product comparisons and review roundups are most exposed. Their whole value was a correct answer to a common question, and an answer engine gives the same answer without the visit. It's the same set of properties that made customer support the first department AI reshaped: repeated questions, answered in text, from a written record (Customer Support Went First). Chegg, whose business was answering homework questions, sued Google over AI Overviews in 2025.

Less exposed, judging from how the mechanism works rather than from measurement:

• Navigational searches, where someone types a brand name to reach its site. There's no question left to answer.
• Transactional searches, where the person wants to buy or book something.
• Communities and first-hand content, which Google says it's sending more traffic to, though nobody has measured that independently.
• Original reporting and data. An answer can repeat a finding, but it needs a source to have one. That protects the originator's standing more than its traffic, since a reader who gets the fact may never visit.

These are the things Where the Moat Moves calls hard to copy.

A publisher's arithmetic

Take an illustrative site: 3 million pageviews a month, 60 percent of them from Google search, earning $20 from advertising for every thousand pageviews (a number the industry calls RPM). That's $60,000 a month.

Two things decide what it loses, and neither is known for any real site in advance. One is exposure: what share of its search traffic comes from queries that now show an AI summary. The other is how much clicks drop on those queries. Take the middle case, 40 percent exposure and the 47 percent click drop Pew measured (from 15 to 8). The site loses about $6,800 a month, 11 percent of its revenue, or about $81,000 a year. Across reasonable assumptions, the loss runs from about 2 percent to about 25 percent.

Scroll sideways to see the whole figure

Now imagine an AI company offers this site a license fee, invented here for illustration. $50,000 a year would cover about 60 percent of the middle-case loss. $25,000 would cover about 30 percent.

Three things follow, and they matter more than the exact numbers. The loss scales with how search-dependent and question-shaped a site is, so the same shift barely touches a site with a loyal audience and badly hurts a search-fed reference site. A flat license fee doesn't move with that exposure, so a fee that rescues one site is trivial to another. And lost pageviews also carried readers who might have subscribed or come back, which this arithmetic leaves out, so it understates the real cost.

Four responses, four different problems

Block. A publisher can tell a crawler to stay away in robots.txt, one crawler name at a time. OpenAI, Anthropic and Perplexity each publish separate crawlers for training, for search and for fetches a user triggers. Blocking the training crawler costs little. Blocking the search crawler removes you from that engine's answers.

Google is the hard case, and it's the crux of the whole issue. Google's documentation says the same crawler rules govern ordinary search and its AI features, and that nothing extra is needed to appear in AI Overviews. A publisher can limit how much text is shown, but keeping a page out of the answers means leaving Google Search. That's exactly what the European Commission opened an antitrust investigation into in December 2025: whether Google used publishers' content for its AI answers without fair compensation while publishers couldn't refuse without losing search. And blocking only works on crawlers that identify themselves; Cloudflare has accused Perplexity of using undeclared ones, which Perplexity disputes.

License. Licensing sells access. OpenAI has signed deals with publishers including the Guardian and Axios, and Amazon with The New York Times, Condé Nast and Hearst. Cloudflare has moved to block AI crawlers by default unless they pay, and is experimenting with payment per use in an answer rather than per page fetched. Licensing works for publishers with archives or audiences an AI company wants. In one survey of publisher leaders, only a fifth expected AI income to be significant. A small site with no one to negotiate with sells nothing.

Litigate. There are two different fights. One asks whether copying text to train a model is lawful; The New York Times's suit against Microsoft and OpenAI is the best known. The other asks whether answering from a publisher's content, bundled with search, is lawful. Chegg and Penske Media, publisher of Rolling Stone and Variety, sued Google under US antitrust law on that theory. On September 30, 2026, a federal judge dismissed both suits for now, finding no plausible claim that Google had traded traffic for content. That closes one US route for the moment. The EU inquiry tests the same conduct under its own rules.

Adapt. Adapting means becoming the source an answer quotes. The research here is thin but suggestive. The paper that named generative engine optimization found that adding quotations, statistics or citations to a page raised its visibility inside generated answers by 30 to 40 percent, while stuffing it with keywords, the old search trick, didn't help. Google says optimizing for its AI features is ordinary search work, with no special markup. The lesson that holds is a content principle: pages with their own numbers, quotes and sourcing give an answer something to cite. Being cited is not the same as being visited, though.

What changes for marketing

For a company selling something, the unit of success moves from rank to mention. A buyer who used to search for the best tool for a ten-person team, scan ten links and open three may now get a shortlist inside the answer.

That has three consequences. Measurement gets worse: Google folds AI features into its ordinary search reports, so a brand can be recommended in answers and see nothing in its analytics. Brand recall matters more: a person reading an answer chooses among names already in mind or named in the answer, and searches for a brand's own name are the ones least touched by summaries. And owned channels gain value: an email list, an app or a community has no intermediary that can summarize it away.

The question underneath

Answer engines summarize pages someone paid to produce. If clicks funded that work and clicks fall, the supply of original reporting, reviews and expert guides could shrink, and answers would rest on thinner, staler sources. That's a hypothesis, not an observation. Much of what answers draw on was never funded by ad clicks: documentation, government pages, papers, forums. And publishers have survived earlier shifts, from answers shown directly on the results page to social networks cutting their referrals.

My judgment: the history counsels against forecasts of collapse, and this shift is still different in kind. Earlier changes cut the amount of traffic but left the structure intact; the link was still the product. Now the product is the text of the answer, built from the publisher's page, and for Google, crawling for search and crawling for answers are one activity a publisher can't refuse separately. The publishers who will be fine are the ones the mechanism predicts: direct audiences, original material, or a service the answer can't perform.

What to do

If you publish, measure your exposure first. What share of your search traffic comes from question-shaped queries? That number, more than any industry average, tells you how much of your revenue is at risk.

Build what an answer can't replace. Original data, first-hand reporting, a tool or calculator, a community. A page whose only value is a correct answer to a common question is now competing with the answer itself.

Own the relationship. Email, apps, subscriptions, events. Every reader you can reach without an intermediary is one an answer engine can't intercept.

Block training crawlers if you want; think hard before blocking search crawlers. And if a license is on the table, compare it with your own exposure, not with what a larger publisher got.

If you market a product, track mentions as well as rank. Treat AI-visibility tools as samples of prompts, not measurements of buyers, and invest in the brand recall that decides which names a reader recognizes in a shortlist (Who Receives the Gain?).

The search engine used to be a middleman that sent readers somewhere else. The answer engine is a middleman that keeps them. Publishers who plan for that will adapt; the ones still waiting for the old clicks to return are planning for a deal that has already changed.

Check yourself

Why does the crawl continue even when the visits fall?

The answer engine still has to read the pages to write the answer. The crawl stays the same size or larger, while the reader has less reason to open the source.

Why is a flat license fee a poor fit for publishers' losses?

The loss scales with how search-dependent and question-shaped a site is, but a flat fee doesn't move with that exposure. It may rescue one site and be trivial to another.

Why is blocking Google harder than blocking other AI crawlers?

Google's rules govern ordinary search and its AI features together. Keeping a page out of the answers means leaving Google Search.