An AI answer costs more to serve than a search
Inference economics, not model quality, decide what AI products can exist. The gap is the whole industry.
By The Signal · · 2 min read
A web search costs a fraction of a cent to serve. A large-model answer to the same question costs multiples of that — the exact multiple is a trade secret, but nobody disputes the direction. That gap, not benchmark scores, is the quiet governor of what AI products get built.
What actually happened
Training a model is a capital expense, paid once and amortised. Inference — answering — is a marginal cost paid on every query, forever. As assistants moved from demo to default, the industry's centre of gravity shifted from training clusters to serving fleets, and the engineering prestige followed.
Who pays, who gains
The bill flows to whoever prices the query. Subscription products cap their exposure with usage limits; free products route hard questions to small models and easy ones to smaller still. When an assistant feels abruptly terser, that is often not a model change but a routing change — cost control wearing a personality.
How it actually works
Serving cost is a function of model size, answer length, and utilisation. The optimisations are plumbing: batching many users through one forward pass; caching shared prefixes; quantising weights to cheaper arithmetic; distilling big models into small ones that handle the common eighty percent. Each technique trades a sliver of quality for a large slice of cost, and the craft is hiding the trade.
What happens next
Purpose-built inference silicon and smaller specialist models keep pushing unit costs down even as usage climbs. The strategic question is whether serving efficiency becomes the moat — models converge, margins diverge. The constraint nobody mentions: latency budgets. Users abandon slow answers, speed costs money, and the cheapest way to serve a query is still to decline it politely.