The price per token is the wrong number

Epoch AI's chart shows the cost of intelligence dropping about 50% per quarter. That's 4× faster than DNA sequencing, 6× faster than compute, 18× faster than lithium batteries. The LocalLLaMA thread around it had people doing the same math I did: at that rate, what costs $100 a month today costs pocket change in a year. The thread's own prediction was that we'd be running coding agents on our phones pretty soon. At 50% per quarter, that's a schedule, not a prediction.

But there's a catch in that curve, and Anthropic's new Opus 5.5 post makes it explicit. You don't buy tokens. You buy finished tasks. A per-token price is a unit cost, and unit costs lie about the total.

Two models can charge the same per token and cost very different amounts on the same task. One reads the code once and fixes it. The other reads it, tries a fix, fails a test, reads again. Every retry is a turn, and every turn resends the conversation so far. The second model bills you for the same context over and over.

The real question is what shape your workload takes, and where in the stack you can cut the waste.

What a task actually costs

A task in Claude Code is a loop. The model reads the conversation, calls a tool, reads the result, and goes around again. Each trip is a request, and four things set what the loop charges:

• Turns. Each turn resends the conversation, so fewer turns means less input billed.
• Cache reads. Most of what gets resent is text from the previous turn, billed at a small fraction of the input price.
• Output tokens. Output costs 5× input, and thinking bills as output.
• Model. The per-token price you pick multiplies everything else.

The example Anthropic runs is a task that starts with 20K tokens of context and grows to 120K as files and tool results stream in. At 40 turns, the average turn sends about 70K tokens. That's 2.8M input tokens billed over the task, while the conversation itself never exceeded 120K. The same task in 25 turns processes 1.75M tokens and bills about a third less in input.

The uncomfortable insight: the cheapest turn is the one you don't need. A retry costs more than any token-saving trick you used to avoid doing the work. That's why the post pushes one habit above all: give the model a way to check its own work. A test to run, a build, a script that calls the endpoint. A model that catches its own mistake on the turn it makes it never pays for the ten-turn failure loop.

Key numbers

• A 40-turn task on a 120K-token conversation bills 2.8M input tokens, though the context never grew past 120K.
• An output token on Opus 5.5 costs 100× a cache read. 60K output tokens run $1.20, the same as reading 6M cached tokens.
• Changing effort or thinking settings clears the cache. The next request pays a full cache write.

Prompt caching is the biggest single lever

The number that should change how you think about sessions: the same 2.8M input tokens cost $11.20 with zero cache hits, $1.62 at a 90% hit rate, and around $0.99 at 96%. No other setting moves input cost that much. A steady session keeps a high hit rate on its own, as long as you don't break the cache.

Opus 5.5 changed the pricing math underneath this. Cache reads dropped from a tenth of the input price to a twentieth, and every other line fell too.

Token type	Opus 5	Opus 5.5	Change
Input	$5.00 / M	$4.00 / M	-20%
Output	$25.00 / M	$20.00 / M	-20%
Cache read	$0.50 / M	$0.20 / M	-60%
Cache read as share of input	10%	5%	-50%

For a long agentic session, most input is cache reads, so the 60% cut lands exactly where it matters most. Anthropic's illustrative session shows the cache line falling from $1.00 to $0.40, the largest drop on the receipt. A short question with a long answer barely benefits, because output dominates. Most Claude Code tasks sit between, with savings from 20% to 60%.

OpenAI is pushing the same direction with GPT-6, which advertises higher cache hit rates, diagnostics for where your cache misses, and explicit breakpoints. Breakpoints matter because a cache matches on the exact prefix: one unstable line at the front of the prompt can invalidate a large, expensive tail. Declaring a stable boundary turns that failure mode into a control.

Quick Take: the biggest cost lever in an agentic session isn't the model's list price. It's how many turns the conversation needs, and whether those turns read from cache.

Output tokens: the expensive half of the bill

Everything above covers input. Output is where the bill actually bites. An output token on Opus 5.5 costs 100× a cache read, and you pay for all of it, including thinking, even when the UI only shows a summary. That's why effort settings move the total so much: effort mostly changes how much the model thinks, and thinking bills as output.

A rough way to price effort: say high effort adds 20K thinking tokens across a task. At Opus 5.5 output prices, that's $0.40. A retry loop of ten turns at 100K of cached context with 10K output tokens in total costs about the same. So high effort pays for itself if it saves one retry, and it's pure waste on tasks medium effort would have finished anyway.

The clearest sign you need more effort is a fix that stops at one layer. A field gets renamed in an API handler. At medium, the model updates the handler, the handler's tests pass, and the client still sends the old field. It did what you asked. It just didn't read far enough to find the second caller. At high, it reads call sites before writing and changes both layers in one pass. But before you raise effort, check whether the model can check its work. A test that exercises the client fails on the exact turn the bad field gets written, at medium, for the cost of one turn.

Model choice moves the bill more than effort, because it sets the price of every token in the session. Anthropic's routing advice for Claude Code: Haiku or Sonnet for lookups, Opus 5.5 for work you supervise closely, and a bigger model like Fable 5.1 for long unsupervised runs. Fable 5.1 lists at $10/M input and $50/M output, two and a half times Opus 5.5, but its cache reads are only $0.25/M, so the gap shrinks on long cache-heavy sessions. Subagents inherit the main model's price unless you pin a model in their definition, and each subagent pays its own tokens while keeping its file reads out of your main context. The tradeoff is real: a small model that misreads a search result sends the main model after the wrong file, and the main model pays for the detour.

Sparsity and quantization: fewer active parameters per token

Price cuts and caching change what you pay per token. Architecture changes how much work an answer needs in the first place.

MiMo-V3 is getting a new architecture, and the core piece, HySparse2, is out today. NVIDIA's Nemotron-Nano-30B-A3B, which the pruning paper uses as a test bed, shows why this direction matters: it interleaves Mamba2, attention, and mixture-of-experts layers in a non-uniform pattern. Sparse and hybrid designs route each token through a fraction of the network. Fewer active parameters per token means cheaper output per unit of quality.

Quantization attacks the same bill from the byte side. Hugging Face just landed native GGUF support in transformers, so you can load a llama.cpp quant directly:

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "unsloth/Qwen3.5-4B-GGUF"
filename = "Qwen3.5-4B-Q4_K_M.gguf"

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    gguf_file=filename,
)

After loading, you're in the normal Transformers APIs. On an M2 Max, the Qwen3.5-4B Q4_K_M hit 70.4 tok/s versus llama.cpp's 71.8, and the 35B-A3B mixture-of-experts variant, which activates about 3B of its 35B parameters per token, ran at 60.2 tok/s. A 4B quant lives comfortably in a laptop, and 60+ tok/s is fast enough for interactive code completion, no perceptible typing delay. Hugging Face's own framing is honest: if you only care about maximum local throughput, llama.cpp is still the better choice. The point is flexibility. You get the small quantized model with PyTorch tooling for debugging, evaluation, and custom generation.

Removing whole blocks instead of shrinking them

The most aggressive cut in this cluster is Multiverse Computing's new pruning paper: deleting entire transformer blocks, not shrinking weights. Block removal turns a 70B model into something that computes like a 35B model, with straight-line latency and memory savings. The hard part is choosing which blocks to delete, because that choice is a many-body problem.

Most existing methods score each block on its own, then remove the least important ones. That's mean-field reasoning, and it misses the couplings. Whether removing block 20 hurts depends on whether you also removed blocks 19 or 24. Multiverse reformulates block selection as a constrained binary optimization problem with a Hessian of pairwise interactions, which maps directly onto an Ising glass, a disordered spin system with a fixed number of up spins. Low-energy configurations of that spin system turn out to be high-performing pruned models. The Hessian is computed once from a small calibration set; after that, evaluating any candidate configuration is a single cheap energy calculation, no model runs, no benchmarks.

The win is at deep compression. On Llama-3.3-70B-Instruct, scored on MMLU without retraining:

Llama-3.3-70B-Instruct	Blocks removed	MMLU
Original	0 / 80	82.2
CBO (Ising approach)	32 / 80	76.6
Block influence baseline	32 / 80	59.3
CBO (Ising approach)	40 / 80	76.9
Block influence baseline	40 / 80	54.0

At 40 of 80 blocks removed, half the model's depth, CBO holds MMLU within about 5 points of the original while the best baseline collapses to the mid-50s. That's a 23-point gap at the deepest setting. At lighter compression the methods are comparable, because couplings only matter once you're cutting deep.

There's a subtlety that pays off even if you never prune a model: the ground state isn't always the best model. For Llama-3.1-8B-Instruct at 16 of 32 blocks removed, most low-energy states cut blocks near the end of the model. But the 17th excited state proposes removing an early block, and after light retraining it beats the ground state on several benchmarks. The common assumption that the best pruning is one consecutive chunk of middle-or-late blocks is wrong, and respecting the coupled structure is what finds the better cut.

What the community is making of this

Reactions across LocalLLaMA and the Hugging Face announcements have been less "this is cool" and more "what do I change on Monday?"

When I loaded the Qwen3.5-4B GGUF in transformers on my M2 Max, the speed parity with llama.cpp was the least interesting part. What sold me was the debugging flow. I could step through generation with standard PyTorch tooling instead of fighting the C++ stack. The 27B UD-Q4_K_M even ran slightly faster in transformers, 15.9 tok/s versus 13.4, which I did not expect.

The cost-of-intelligence thread had a different energy. People argued about whether 50% per quarter is sustainable, and the responses split between "the curve has to flatten eventually" and "the price of a fixed capability has been dropping this fast for years." My read: the capability you can rent for $100 a month today is moving down the price curve so fast that the profitable habit is to re-price your workloads quarterly. What was a borderline-priced batch job in Q1 is cheap enough to run interactively by Q3.

The Ising pruning paper is getting attention because it composes with everything else. Block removal stacks with quantization, SVD, and width pruning instead of competing with them. Remove 40 of 80 blocks while holding quality near the original, then quantize what's left, and the combined cost curve gets steep quickly.

Common Pitfalls

Don't change effort mid-session if you care about the cache

Effort settings are part of the prompt the cache matches on. Changing them clears the cached conversation, and the next request pays the write price on the whole context. On API keys the cache lifetime is five minutes, so a six-minute coffee break turns a $0.02 read into a $0.60 write. Check the cache durability before you step away from a long session.

Comparing models on price per token instead of price per task

When I compared Opus 5 and Opus 5.5 on list price, I assumed a flat 20% saving. A cache-heavy session saves up to 60%; a short question with a long answer saves closer to 20%. The spread is entirely task shape. If you're choosing between models, run the same task on both and compare turns, output tokens, and cache reads. /usage in Claude Code gives you the raw numbers.

Treating pruning as independent block scoring

Scoring each block alone and deleting the worst ones works at light compression and fails hard when you cut deep. The 23-point MMLU gap at 50% compression is the cost of ignoring couplings between blocks. If you prune, search over combinations, not individual blocks.

Expecting transformers to replace llama.cpp for raw speed

Native GGUF support is for flexibility, not throughput records. On Apple Silicon the ggml kernels get you near parity, but if your only goal is maximum token speed on a quantized model, llama.cpp still wins. Use transformers when you need PyTorch tooling, debugging, or custom generation loops.

Optimizing tokens while ignoring the retry cost

Every token-saving habit, lower effort, smaller model, less context, can cost you a finished task. A retry loop costs more than the savings. Also audit your prompts: instructions written for an older model can make a new one write more and repeat tool calls. In an internal migration benchmark, moving from Opus 4.8 to Opus 5.5 at low effort cut cost by 18%, and a prompt audit cut it another 9%. The audit removed ritual instructions, a mandatory six-step procedure, a scratchpad rule, a verify-twice rule, none of which helped.

One Thing to Remember: the unit that matters is cost per finished task, not cost per token. Caching, sparsity, quantization, and pruning all reduce that number, but only if you measure your actual workload shape first.

The practical takeaway

If you run agentic workloads like Claude Code, measure turns and cache-hit rate before touching anything else. The 60% cache-read price cut on Opus 5.5 is free money only for sessions that keep a steady prefix, and it's worth zero for short, uncached queries.

If you're a local-inference user, move your everyday models to GGUF