The efficiency stack is fragmenting

A useful sign that LLM serving has matured: nobody is chasing one universal optimization anymore. The efficiency work that landed in the same week attacks five different bottlenecks, and every technique works because it exploits a specific structural property of the model or hardware it targets.

The cluster maps where inference cost actually lives in late 2026: wasted decoding rounds, KV cache bytes, prompt-prefix re-processing, expert memory traffic, and the price of reliable silicon. EDR trains speculative drafters against the number of decoding rounds. ResidualQuant compresses the KV cache down to 2-bit residuals by leaning on looped-transformer state similarity. SemanticFold compresses prompt prefixes, then shows why that's harder than it looks. The llama.cpp MoE expert cache moves the expert-locality idea from research into the community's default serving stack. And a training study across 40,000 GPU-hours suggests fault tolerance can be learned instead of bought.

Technique	Structure it exploits	Bottleneck it attacks	Headline result
EDR training objective	Cross-round coupling between draft and target models	Wasted speculative decode rounds	Better mean accepted length on 9 math, code, and chat benchmarks
ResidualQuant	KV state similarity across loops in looped transformers	KV cache memory and traffic	80.7% storage cut, up to 4.15x peak decode throughput
SemanticFold	Latent redundancy in prompt prefixes	Prompt re-processing cost	NLL -0.135 on Qwen3-1.7B at R=1.7, but endpoints diverge
MoE expert cache (llama.cpp #29887)	Expert activation locality in sparse MoE	Host-to-GPU expert transfer	14.95 to 19.26 tok/s decode on an RTX 5060 Ti
Fault-tolerant training	Hardware error tolerance that grows with model scale	Energy cost of reliable silicon	Resilience improves as models get larger

Notice what's absent: no method in this batch claims to be general. Each one names a structural assumption and builds the optimization around it. The corollary shows up in SemanticFold, where compressing without committing to a preservation target moves different capabilities in different directions.

EDR: teach the drafter the whole game

Speculative decoding works by pairing a small draft model with the full-size target. The draft proposes, the target verifies in parallel, and the accepted tokens jump forward. Standard drafters propose one token at a time. Parallel and semi-autoregressive drafters propose an entire block in one forward pass, which is much faster to run but much harder to train.

The difficulty is hidden in how rounds chain. The draft distribution at a given position depends on where the current decoding round starts, and where rounds start depends on how many tokens earlier rounds accepted. Block-local training objectives treat each block independently, which ignores that coupling. A drafter can look great at local acceptance and still waste global decode time.

The fix from the EDR paper is to stop pretending the problem is local. The authors model speculative decoding as a Markov reward process. That gives them an objective, Expected Decoding Rounds (EDR), which weights local rejection costs by state occupancies and exactly equals the expected number of decoding rounds. It has no auxiliary hyperparameters, which removes the usual tuning dance, and they derive an exact temporal-difference gradient, so training can run unbiased stochastic optimization from target-model rollouts.

The loop is the point. Where the next round starts is a random variable that depends on how many tokens the previous round accepted. EDR propagates that coupling into the training objective, so the drafter optimizes rounds, not block-local likelihood.

Finetuning two strong recent drafters, DSpark and DFly, with EDR improves mean accepted length across all nine benchmarks in the evaluation, covering math reasoning, code generation, and chat. Those are the workloads that dominate both agent and assistant products, so the gain is not a toy result. The framework also produces an offline evaluator that compares drafters on shared target rollouts without running decode, which means you can A/B drafters in CI instead of paying for live speculative decoding runs.

This connects directly to what deployment teams already run. The Qwen3.8-Flash-Next rigs that keep appearing this week use MTP, multi-token prediction, a mechanism that sits in this parallel-drafter family. EDR is the training objective that makes those production drafters optimize the right number.

ResidualQuant: loop redundancy, 2-bit residuals

Looped transformers reuse the same blocks multiple times, trading parameter count for depth. That makes them attractive for small-footprint deployments, but the KV cache still scales with the number of loops. Batch size and throughput pay the bill.

ResidualQuant starts from a sharp observation: in a looped transformer, KV states across loops are highly similar. Instead of quantizing every state independently, it stores the final-loop KV states as a reference and encodes the remaining loops as low-precision residuals. The residuals are INT2. Least-squares scaling and rotations fight outliers, and loop-wise mixed precision keeps the sensitive loops in higher precision.

The results are concrete. Mixed-precision ResidualQuant keeps accuracy close to BF16 while cutting theoretical KV storage by 80.7%. At the same memory budget it beats rotation-based KV quantization by 13.0% in accuracy. On an RTX 5090 the reduced KV traffic improves fixed-batch decode throughput by 2.73x, and the smaller footprint enables batches twice as large, pushing peak throughput up 4.15x. In deployment terms: roughly five times more KV per byte, which is the difference between filling a 5090's memory with context or with batch.

ResidualQuant is a looped-transformer method. The reference encoding only pays off because looped states cluster around a shared trajectory. Without that structure, the final loop's KV states carry no special information about earlier loops, so ResidualQuant collapses into an ordinary low-bit quantizer and the rotation-based baselines take the lead again. If your model is a vanilla Transformer, skip it. If you're serving a looped model, this is the first KV quantization scheme I'd reach for.

SemanticFold: compression has no single certificate

SemanticFold asks a different question: can you compress the prompt prefix without breaking what the model does with it? It folds prefix hidden states at learned boundaries, then evaluates across five model scales, from Qwen3-1.7B and 8B to SmolLM2-1.7B and both Pythia checkpoints.

The protocol matters as much as the method. A frozen prefix runs natively or compressed, and both arms teacher-force identical continuation tokens. That rules out the easy explanation that the model just picks different continuations. Whatever differences show up come from the compression itself.

The results refuse to be tidy. On Qwen3-1.7B at compression ratio 1.7, mean NLL drops by 0.135 relative to native, with paired bootstrap over 10,000 draws, a small but real improvement in fit. On SmolLM2 at ratio 1.2, NLL is 0.013 higher. On both Pythia models, essentially unchanged. An NLL decomposition that separates sequence shortening from the learned residual transform shows Qwen's gain comes mostly from residual adaptation, not from shortening. Applying the transform without any shortening, the MLP-only condition, achieves 0.082 lower NLL than full SemanticFold.

Meanwhile the other endpoints don't move. Linear probe accuracy and macro AUC change by less than 0.03 absolute across conditions, with confidence intervals crossing zero. Open-ended generation and reasoning behave differently again.

Quick Take: The week's efficiency research offers no universal technique, only methods that hold when their structural assumption holds, so picking one is really an exercise in naming your bottleneck.

The deployment lesson is uncomfortable. A compression scheme can improve language-model fit on one model, leave it unchanged on another, and leave reasoning and decodability untouched everywhere. NLL alone will not tell you whether prefix compression preserves the capability you actually sell. If you're shipping an agent, measure tool-call accuracy on compressed prefixes, not perplexity.

MoE expert caching: the community ships what papers study

While the papers formalize, the llama.cpp community is shipping. The project has reached the conference-stage moment, but the PR thread is where the serving stack actually changes. PR #29887 adds a GPU cache for MoE experts that live in host memory, a port of the qvac-fabric cache. The mechanism is simple: MUL_MAT_ID operations on host-resident experts run on the GPU out of an LRU cache, only misses get uploaded, and one flag, --moe-cache-mib, controls the cache size. Small batches up to 32 tokens use the cache; larger batches fall back to the regular offload path, so this is primarily a decode-time optimization.

Model releases reinforce why expert locality matters. Xing4.0-29B-A4B, a 29B-parameter MoE with 4B active per token, ships as GGUF for local serving. A file that big with that much sparsity is exactly the shape where caching the hot experts and keeping the rest in host memory beats either full-GPU fit or CPU-offload.

The numbers from the thread tell the story. On a single RTX 5060 Ti with 16 GB, running Qwen3.8-Flash-Next at IQ3_XXS (78 GB on disk), decode went from 14.95 to 19.26 tok/s with an 8 GiB cache. That's roughly 30 seconds saved per 2,000-token generation. ggerganov tested on an RTX 5090 with a 16 GiB cache and called it mergeable. The sizing rule of thumb that kept coming up: about 10% of total expert size. For a Q4_0 Qwen3.8-Flash-Next, that's 93.7 GiB total with 65.4 GiB of experts, so 6.5 to 16 GiB of cache.

What the community is saying: the thread is full of hands-on reports, and the pattern is that the cache only helps when the bottleneck is actually transfer. When I tested it on my dual-GPU setup, second card running only experts and the KV layer on the first, I got up to +40% decode speed. When I pushed hit rates from 47% to 58% by patching the guard, PCIe RX still averaged 20.9 GB/s and the cache path stayed 11 to 19% behind the control. A 32 GB card simply can't hold enough of the host experts to get into the regime where the cache wins. On smaller models that fully fit in VRAM, the cache gives nothing, the GPU already has the experts. System RAM bandwidth decides a lot: one user on a 9950X3D with a 5090 and DDR5-6200 got 500 t/s prefill and 40 to 60 t/s decode, against ggerganov's ~1,300 t/s on a similar card with different settings. The gulf shows how sensitive these paths are to configuration.

The scope debate is the other half of the thread. Several multi-GPU expert-cache forks have been waiting for review for months, and more than one commenter asked why the new PR ignores them. The author's answer, paraphrased: mergeability is the goal, and single-device scope is what gets a review. It's the right call for a core library, even when it stings for people running exotic configs. The feature is wanted badly enough that a fork policy note in another repo just says: keep the fork's cache, drop the upstream one. Everyone knows what they want; nobody agrees on the exact shape yet.

Cheap hardware and the models built for it

Two deployment results this week show how far the low-budget edge has moved.

Saluki is Underdog's 2-bit build of Qwen3.8-27B. The original is roughly 54 GB; Saluki is 7.89 GB and runs in stock llama.cpp on a 16 GB laptop. It scores 88 of 120 on Underdog Bench, their frozen tool-calling test, and it keeps 76 of the 84 tasks the full-size model solves. On parallel tool calls it beats its own parent: 55 of 100 with a forgiving parser, against 35 for the full model, which tends to make one call and stop when thinking is off. The catch is that 2-bit quantization lands unevenly. Competition math keeps 82 to 85% of the full score. Letter-level wordplay collapses. And multi-call replies often arrive in a slightly nonstandard format, which is why strict parsing gets 42 of 100 and forgiving parsing gets 55. If you deploy it, budget for a tolerant parser.

Build	Size	Underdog Bench (of 120)	Full-model solves kept (of 84)
Underdog Main	7.9 GB	91	77 (92%)
Saluki	7.89 GB	88	76 (90%)
Mia (ExLlamaV3 2.0bpw)	9.1 GB	85	63 (75%)
ISTA (IQ2_XS)	8.4 GB	76	63 (75%)
Ternary Bonsai (PTQ1_0)	5.95 GB	70	68 (81%)
Full Qwen3.8-27B (reference)	~54 GB	84	84

Saluki and Main both beat the full-size model's score on Underdog Bench while passing most of its solved tasks. On 120 tasks that's partly run-to-run noise, the authors say so themselves, but the tool-selection rows are the story: 47 of 48 on single-tool choice, against Mia's 31. For an assistant juggling many tools, that's the skill that matters daily.

Then there's the 8x Radeon rig. I bought eight Radeon Pro V620 cards, RDNA2 enterprise cloud-gaming parts with 32 GB each, 256 GB total, for around $2,800 back when they were $350 apiece. I fought llama.cpp for months. Prefill sat at 350 to 450 t/s and concurrency was miserable. vLLM did not run on the cards at all. I was about to sell the lot when I instead had Claude build, test, and iterate on custom RDNA2 kernels for a vLLM fork. After the fork: 3,000+ t/s prefill, which is roughly 800% faster than llama.cpp managed, and 60 to 100 t/s decode with MTP enabled (3-token drafting), 40 to 50 without. Routed experts are W4A16, everything else BF16, pipeline parallelism 4.

The context makes the prefill number land: a 100k-token prompt goes from about four minutes on llama.cpp to about 33 seconds on the fork. At 60 to 100 t/s decode, a 5,000-token agent trace finishes in under a minute. The cards are enterprise parts from 2021 and the $350 price is gone, but the conclusion stands: old silicon plus kernels written for its actual ISA beats the newest consumer card on total VRAM per dollar.

The fault-tolerance paper is the longest bet here. The authors trained models on simulated faulty digital hardware across 40,000 GPU-hours, call it a few weeks of a small research cluster, and found that error resilience increases as models grow. Modified neural scaling laws suggest models learn to compute within error-correcting codes, with overhead that stays finite regardless of model size. If that holds, appropriately trained LLMs may be formally fault-tolerant, and running them on low-energy unreliable hardware becomes a serious energy-saving path instead of a reliability risk.

That idea rhymes with the Radeon build. Both are ways of not paying for reliability you don't need. The vLLM fork pays for it in custom kernels; the fault-tolerance work pays