The fine print in Sonnet 5's new tokenizer

Whenever an artificial intelligence company announces that its new model is cheaper, it is worth taking a minute before applauding. Not because the headline is necessarily false, but because it is often built on a definition of “cheap” that only works if nobody does the full math. The price per token is just one variable in an equation with more unknowns than can fit in an announcement paragraph.

Anthropic has just launched Claude Sonnet 5, promising more capability at a lower cost. And, as has become customary in this industry, the fine print is buried one layer deeper, hidden in a technical detail that rarely appears at the top of the press release: exactly how the tokens that are now cheaper are counted.

The headline: More agentic, cheaper?

The announcement itself is ambitious even by the already inflated standards of the sector. Anthropic presents Sonnet 5 as “the most agentic Sonnet to date” (at some point someone should ask whether the formula “the most [adjective] [noun] to date” has become too obvious to keep using, after Apple perfected it for selling devices to users who generally do not want them to get worse), capable of planning and using tools autonomously on tasks that previously required Opus, its high-end model with a very different price. The claimed improvements are not cosmetic: better agentic search, better handling of graphical interfaces, fewer hallucinations, less sycophancy, and stronger resistance to prompt-injection attempts.

The launch price matches that ambition with apparent generosity: $2 per million input tokens and $10 per million output tokens until August 31, 2026, compared with Sonnet 4.6’s $3/$15. One third less, on paper and for now. Anthropic is explicit in how it positions the model: at medium effort, it would offer “substantially improved cost efficiency”; at high effort, performance able to match Opus 4.8 itself.

On paper, it is a neat announcement: a smarter, more autonomous, cheaper model, all three ideas in the same paragraph. The problem is that those three claims do not live in the same paragraph of the technical catalog. And that is exactly where the arithmetic starts getting complicated.

The fine print: The tokenizer that absorbs the discount

The first piece of fine print is not in the tokenizer, but in the announcement calendar itself. The advertised costs are not “the price” of Sonnet 5: they are Sonnet 5’s promotional price, valid until August 31, 2026. After that date, the standard price becomes $3 per million input tokens and $15 per million output tokens, exactly the same amount Sonnet 4.6 already charged. There is no permanent discount in Anthropic’s catalog. There is a launch promotion with an expiration date, and an underlying price that has not gone down by a single cent.

On top of that calendar sits the second variable: the new tokenizer introduced with Sonnet 5. For the same input text, it produces between 0% and 35% more tokens than Sonnet 4.6’s tokenizer, with a reasonable midpoint around 30%, depending on the type of content. Even if each token costs less, more of them are needed to say the same thing. What the headline does not answer is what happens when those two numbers are combined.

The promotion softens that increase while it lasts. When it ends, it stops doing so.

Seen this way, the promotion starts to look less like a simple commercial gesture and more like an adoption window: just enough time for the model to become integrated into workflows, turn into the default choice across half an application, and become too inconvenient to replace when, in September, the invoice starts reflecting the real cost of the fine print.

We already saw this with Opus 4.7

None of this is an anomaly specific to this launch. When Anthropic updated the tokenizer with Opus 4.7, the pattern was the same: the price per token barely changed on paper, while real invoices, according to independent measurements, grew by what could reasonably be described as a double-digit increase for many workloads. The same text, tokenized more finely, ended up costing more even though nobody had touched the posted rate.

It is true that not everyone felt the hit equally. Teams that were already caching much of their prompts, reusing context between requests instead of sending it from scratch each time, probably absorbed part of that increase, since tokens served from cache are billed at a substantial discount. But caching does not implement itself and is not universal, and relying on it to make the economics work is, once again, a condition that does not appear in the headline.

When the cheaper model does the expensive model’s job

There is another part of the announcement that deserves more attention than it is getting: Anthropic is not selling Sonnet 5 only as a cheaper version of itself, but as a partial substitute for Opus. At medium effort, it promises “substantially improved” cost efficiency; at high effort, performance capable of matching Opus 4.8 itself. Translation: the cheaper tier is starting to do the work that previously justified paying for the expensive tier.

With a bit of distance, that approach can be read as more than a product improvement. With Haiku, Sonnet, Opus, and now Fable, Anthropic’s catalog has moved from three easy-to-explain tiers - cheap, balanced, powerful - to four, with Fable occupying the step that used to belong to Opus and leaving Opus in an increasingly hard-to-justify middle position. Sonnet getting so close to Opus may not be just a technical coincidence, but the first step toward a more natural three-level structure again - Haiku, Sonnet, Fable - in which Opus eventually dissolves into the tier it now claims to match.

Whether or not that is intentional, the immediate effect is the same for anyone designing systems on top of these models: if the middle tier covers much of what used to require the premium tier, the convenience of choosing a model by catalog category rather than by task starts to break down. The discipline ahead, as I already argued in another article, looks more like cost engineering than prompt engineering: deciding which model, with which effort level and what context budget, instead of assuming the model name comes with the decision already made.

What does not match Opus quite so easily is safety. The announcement itself admits that Sonnet 5 still trails Opus 4.8, and Mythos Preview, in behavioral audits, and ships with cybersecurity safeguards enabled by default because, unlike general capability, this is not yet an area where there is parity to sell. “Almost Opus” can be a generous description when applied to what the model does well, and much less generous when applied to what it still does not fully control.

None of this makes Sonnet 5 a bad model, or the promotion a deception: it is simply a promotion, with the limits and expiration date of any other. And there is always the option to stay with older models, at least until they are marked obsolete and retired. The pattern repeats often enough - tokenizers change, hierarchies blur - to make it worth reading the next headline with the calculator open. The fine print in this industry keeps growing; the only thing that changes is which paragraph it chooses to hide in this time.