The bill shows up as cache traffic

Every efficient-inference paper I read this month lands on the same wire. Not the weight file. Not the FLOP count. The KV cache.

Run the arithmetic for one decode step. Shared weights are read once and amortized across whatever batch fits. The history is private. Read it for request A, then again for B, then again for C. Batch size forty means forty separate histories pulled through the memory system. Double the context length and the private read doubles with it. That's why a 128K serving bill doesn't look like an 8K one at identical token throughput.

The Token-Energy Law paper gives this shape. Two coordinates: active bytes per output token, and realized decode time. Active bytes split into dense weights shared within an iteration and private KV histories read and updated per request. Combine bytes and time in one affine form, calibrate two coefficients per model and hardware config, and you get joules per generated token.

That framing is the spine of everything below.

Four of the six papers shrink the bytes. One cuts the sequential steps. One argues the whole problem is data movement wearing a math costume.

Let the agent steer its own cache

The standard fix when an agent trajectory outgrows the window is trimming or summarization. Both rewrite the prefix. Rewriting the prefix invalidates every cached state behind it, so you re-prefill the retained context and pay the cost you were trying to dodge.

I've watched a team hit exactly this wall. They compressed a 60K tool-call trajectory down to 12K, cut their bill by a third, and lost four points on their internal eval. The re-prefill they were avoiding came back anyway, because each new summary invalidated the cache behind it.

Cache Engineering for Agents proposes a different lever. Two operations. drop excludes selected entries from future attention. repos adjusts the position encodings of keys you keep, so retained context stays coherent after something ahead of it disappears. The trajectory itself is never modified.

The serving consequence is the part worth internalizing. History dependence: two identical token sequences can produce different cache states if the operations that led there differed. Hashing the prompt no longer identifies a reusable cache. Prefix matching is insufficient, full stop.

AKV, the accompanying engine, keeps a stateless interface on the outside (existing serving APIs work unchanged) and manages stateful reuse, eviction, and restoration inside. A tool-response rolling-drop policy gains 1.7 points on average benchmark score while cutting average cost 27.7% versus threshold-triggered summarization. On the serving side, 1.2x output throughput and 1.5x faster P95 latency. Letting the policy improve recursively adds another 7.1% cost reduction over the hand-designed version.

Both moved in the right direction at once, which is unusual. Efficiency work normally buys cost with accuracy. Here the thing being dropped is stale tool output, not reasoning, so the eval improves because the context gets cleaner.

Quick Take: six papers, one conclusion. The KV cache is the cost center, and the useful work now is deciding which bytes get read, how many dimensions they carry, and how long they stay resident.

Predicting several tokens without a second model

Ultra-low-bit backbones cut what you have to store and what you have to move. The linear layers get cheaper too. Multi-token prediction cuts sequential decode steps by drafting several tokens in parallel. Combining them sounds free and isn't.

BitMTP spells out why. Masked drafting gets no intermediate token inputs, unlike next-token training. Wider computation and causal KV updates for the whole drafted block can eat the savings from skipping steps.

The fix is a joint next-token plus MTP objective on one model, giving it both autoregressive generation and block-causal drafting from the same weights. Three modes (confidence-only, speculative-only, hybrid) give you quality-throughput operating points without a separate draft model and verifier. Accepted-prefix alignment keeps the committed prefix, the materialized causal KV boundary, and the CUDA Graph replay state consistent, which is the plumbing that makes continued generation after a partial accept actually correct.

On an H20: 2.2x five-task mean speedup on BitNet b1.58-2B, and 2.1x to 2.4x on Bonsai models from 1.7B to 4B at comparable quality. On RTX 3090 and Jetson Orin, up to 2.0x five-task mean throughput and 3.3x on MBPP+ for Orin.

A 2.2x on a 2B backbone at equal quality means a 30 tokens/s stream becomes roughly 66 on the same card. The Orin number is the more interesting one. 3.3x on Python generation is the gap between a local coding assistant that demos well and one you'd actually ship to a field device.

Half the dimensions, none of the reconstruction

Most head-dimension compression stores fewer dimensions and then rebuilds full-dimensional representations at inference. You saved memory and spent FLOPs. RACE goes after the second half of that trade.

Three constraints on the projection: apply a shared projection to queries and keys so they stay in a common reduced space, derive it jointly from query and key activations, and stay compatible with the pretrained RoPE transform. Restricting the reduction to coordinate selection gives the clean result: RoPE compatibility holds if and only if complete rotary pairs survive. That turns an architecture search into a selection problem with a checkable condition.

RACE picks rotary pairs by joint query-key activation energy, and the authors prove that criterion minimizes a separable upper bound on post-RoPE query-key product approximation error. Because it's coordinate selection, the reduction folds into the weights. No runtime reconstruction. No custom kernels.

At 50% KV retention and 256K context: 47.8% fewer total inference FLOPs, 1.79x throughput, 43.2% lower time-to-first-token, 28.1% less allocated GPU memory, measured against an uncompressed model. Competitive with reconstruction-based methods on language modeling, zero-shot, and long-context tasks, and ahead of structured pruning.

One caveat, stated plainly: that 1.79x is against an uncompressed baseline. Your stack is probably already quantized, so the delta against what you run today will be smaller. Still real, smaller.

Sparsity that keeps most of the mass

Top-k retrieval is deterministic, catches the high scorers, and throws away residual attention mass. Sampling covers more ground with higher variance. Hybrids combine top-k with uniform sampling and still leave a visible gap to dense attention at aggressive sparsity.

Stratified Sparse Attention attends deterministically to the top scorers, samples the residual using proxy attention scores, and corrects for inclusion probabilities so the recovered mass stays unbiased. Top-k, top-p, and top-k-plus-sampling all fall out as special cases of the same estimator.

Five models from 3B to 27B, four benchmarks. On RULER-HARD with Ministral-3-8B at 50x attention sparsity, accuracy goes from 81.8% to 84.2% at 32K context and from 63.8% to 69.3% at 128K. That's 94% and 85% of dense-attention performance respectively. Optimized CUDA kernels give 2.96x decode speedup over FlashInfer at 1M context.

50x sparsity at 128K means attending to roughly 2.5K of 128K entries. Recovering 85% of dense at that ratio is the whole argument. At 32K the recovery is 94%, but the absolute gain over the best baseline is 2.4 points, so use SSA there for throughput, not accuracy.

Key numbers, pulled from all six papers 27.7% lower average cost for a rolling-drop cache policy, against threshold-triggered summarization 1.2x output throughput and 1.5x faster P95 latency from the AKV engine 47.8% fewer inference FLOPs at 50% KV retention and 256K context, buying 1.79x throughput 81.8% to 84.2% on RULER-HARD at 32K under 50x sparsity, and 63.8% to 69.3% at 128K 3.73% worst-case error on held-out energy estimates from the Token-Energy Law

Joules per token is a configuration, not a property

Here's where the energy paper earns its place. It doesn't propose an optimization. It proposes a measurement model you can plan against.

Three models, BF16 decoding, NVIDIA Hopper and Blackwell, including two-GPU tensor parallelism on Hopper. Fixed coefficients, held-out condition-mean energy estimates within 3.73% of measurement. That's tight enough to put in a spreadsheet instead of a benchmark harness.

What the model tells you is that batching amortizes the weight term and does nothing for the private KV term. Longer attended histories raise private bytes. Throughput sets the time coefficient. You can't batch your way out of a long-context energy bill. Twenty requests on 128K histories cost roughly twenty private reads no matter how you schedule them.

Local long context is a data movement problem

The local-inference paper asks a narrower question than the usual "you need quantization to fit on one GPU" framing. It asks what limits long-context serving for a single user once capacity isn't the binding constraint.

The study runs models across several GPU generations rather than characterizing one model on one card, which is what makes the findings general. Self-attention dominates runtime at long context, as everyone expected. The surprise: the current structure of self-attention leaves key GPU resources significantly underutilized, and the newest GPU is often the least well served by it. Paying for a newer card can make the utilization picture worse, because the attention loop can't feed it.

The complaint I keep hearing from people running models on their own hardware isn't about tokens per second. It's about the fan spinning up at 4K context while the GPU sits mostly idle. This paper puts a name on that, then reframes it as data management: paging the KV cache, cache utilization, synchronization inside the GPU. The authors brought it to the data systems community on purpose, since none of those are new problems in that field.

Take the reframing seriously. If your bottleneck is a memory hierarchy your attention kernel doesn't understand, buying bandwidth won't fix it, and neither will another 10% off the weight file.

All six, side by side

Two of the six don't speed anything up at all. The energy paper and the local-inference study are measurement work, and they're the reason the other four are interpretable. Everything else attacks a different coordinate of the same cost surface.

Paper	What it reduces	Headline result	What it costs you
AKV	entries retained in the active cache	27.7% lower cost, 1.2x throughput	prefix caching needs history awareness
BitMTP	sequential decode steps	2.2x on H20, 3.3x on Orin MBPP+	needs a low-bit backbone to start from
RACE	dimensions per cached entry	1.79x throughput, 43.2% lower TTFT at 256K	measured only against an uncompressed model
SSA	entries attended per step	2.96x decode at 1M context	custom kernels, thin gains below 32K
Token-Energy Law	nothing, it predicts	3.73% worst-case held-out error	recalibration per hardware config
Local long context	nothing, it diagnoses	new GPUs left underutilized	findings, no artifact

The four optimizations compose poorly in practice. AKV changes what's resident, RACE changes what each resident entry costs, SSA changes how many entries you touch, BitMTP changes how many passes you make. Stack all four and your profiling story gets ugly fast, because each one moves the bottleneck somewhere else. Pick two, measure, then pick a third.

What trips people up

Reusing a prefix cache after you mutated the prefix

If any part of your stack drops or repositions context, a prompt hash no longer identifies a valid cache state. History dependence means identical tokens can carry different attention semantics. Audit your reuse path before you turn on any cache manipulation, or you'll get subtly wrong outputs with great latency numbers.

Measuring compression against the uncompressed baseline

RACE's 47.8% FLOPs cut is against a full-precision model. Your production stack is already quantized or already using paged attention. Re-baseline against the config you actually deploy, or you'll budget for a speedup that arrives at half size.

Buying sparsity for a workload that never exceeds 32K

SSA recovers 94% of dense at 32K and 85% at 128K, but the payoff curve is steep in the wrong direction for short contexts. At 8K you're paying kernel complexity for a rounding error in latency. Sparsity is a long-context tool.

Bolting multi-token prediction onto a stack that already has a draft model

BitMTP's advantage is replacing the draft-verifier pair with one model and a joint objective. Keep the separate draft model and you inherit the wider matmuls plus causal KV updates for the drafted block without the memory savings.

Treating joules per token as a model constant

It isn't. The coefficients move when you change tensor-parallel degree, batch size, GPU generation, or history length. Recalibrate after any infra change, or your energy forecast is just a number with a decimal point.

One thing to remember

The KV cache is where four independent research groups converged, and they converged because that's where the bytes are. Weight compression is a solved-enough problem that a 2B model fits in a coffee cup. Context is not, and context is what agents, codebases, and document workflows actually consume. If you take one thing from this batch, take the accounting: bytes per output token, split into shared and private, multiplied by realized decode time. Everything else is an implementation detail of which term you attack first.

What to do with this

If you run agents with long tool-call trajectories, adopt cache-level control before you reach for summarization. AKV's numbers (1.7 points of accuracy and 27.7% cost, both in your favor) exist because dropping stale tool output is strictly better than rewriting the prefix and eating the re-prefill. The catch is that you have to build history-aware cache reuse, which summarization lets you skip.

If you serve 128K or more and can't afford accuracy loss, pair SSA with RACE. SSA keeps you within 85% of dense attention at 50x sparsity on 128K inputs, and RACE cuts time-to-first-token 43.2% by halving the width of every cached entry. Below 32K, skip SSA entirely and take RACE alone, since the sparsity payoff isn't there yet.

If you're budgeting energy or shipping to edge hardware, calibrate the Token-Energy Law on your own config first (3.73% error makes it a planning tool, not a guess), then put BitMTP on the roadmap. A 3.3x on MBPP+ for a Jetson Orin changes what a local coding assistant can be, and that hardware class is where the next round of deployment fights will happen.