The decoding bottleneck

Ask any serving engineer what limits LLM throughput and they'll point at the same stage: decoding. Prefill is compute-bound and parallel-friendly. Decoding is memory-bound, one token at a time, and every token costs a full pass over the model.

Five papers landed in the last few weeks, and each attacks a different piece of that per-token pass. SparseDecoding prunes weights the way they're actually read during generation. VFold compresses the value cache by exploiting redundancy across layers. TokenRouter routes individual tokens to cheaper models mid-response. DLoop makes speculative decoding verify less often. And reViT, from computer vision, shows a single transformer block can serve every depth, cutting stored parameters by roughly 70%.

None is a silver bullet. But together, they map the remaining waste in LLM serving.

Why a token costs a full weight pass

The arithmetic sets the floor. An 8B-parameter model at fp16 stores about 16 GB. On an A100-class GPU with roughly 2 TB/s of HBM bandwidth, one full read takes about 8 ms. That's the minimum time to generate one token, before attention, sampling, or kernel overhead. No amount of GPU flops changes it, because the bottleneck is bytes moved, not compute.

The KV cache adds a second, growing bill. At long contexts it can rival the weights per request, and it competes directly with batch size. Compress it, and you serve longer contexts or bigger batches on the same memory.

Here's where the five papers intervene:

SparseDecoding: calibrate pruning on generated tokens

Pruning for LLM inference has a blind spot. Hessian-guided methods pick which weights to drop using a calibration set, usually natural text collected in advance. But at decode time the model consumes its own output. SparseDecoding shows this mismatch is real: the Hessian over natural sequences differs from the Hessian over generated sequences, and the activation distribution during generation drifts away from what pruning assumed. The result is a pruned model that looks fine on calibration loss and degrades on long generation.

The fix is principled and slightly awkward to implement: collect calibration activations during the dense model's own autoregressive generation, and exclude prefill entirely. That aligns the pruning objective with the data the model will actually see at serving time.

The system side matters just as much. Most pruning work that reports real speedups optimizes sparse matrix-matrix kernels (SpMM), which suit prefill-style workloads. Decoding is a sparse matrix-vector problem (SpMV): one token, one vector, over a big sparse weight matrix. SparseDecoding ships an N:M sparse matrix-vector kernel with bitmask indexing and fixed-step traversal, so 2:4 pruning translates into fewer bytes read per step rather than a kernel that's fast on paper and slow in practice.

The headline: up to 1.48x end-to-end wall-clock decoding speedup on A100 GPUs across Llama-3.1-8B, Llama-3.3-70B, Qwen3-14B, and Qwen3-32B. To put that in concrete terms, a decode that ran at 50 tokens per second runs at about 74. It's not a 2x dream, but it's end to end, measured, and it beats fixed-text calibration on long-form generation benchmarks. The accuracy cost is nonzero; this is pruning, not a free lunch. But it's the cheapest lunch on this menu, because it needs no architecture change and no serving redesign.

VFold: the value cache repeats itself

KV cache compression usually means two things: quantize the cache to fewer bits per entry, or prune entries (mostly keys) that look unimportant. VFold finds a third source of waste, one that sits in plain sight: the value cache is redundant across layers.

Different layers store versions of the same values, transformed step by step. VFold exploits that symmetry to merge values across layers, and the design choices are what make it practical. It's a post-hoc merge: no architectural changes to the model, and no harmful performance degradation in their tests. It also composes. Stack it on top of high-ratio quantization or key cache pruning, and you reach compression ratios that neither method reaches alone.

That composition point is the one to hold onto. Cache compression methods usually compete with each other. VFold is built to multiply.

Quick take: the biggest wins come from matching your optimization to the memory traffic of decode, not the compute of prefill.

TokenRouter: don't pay flagship prices for easy tokens

Model routing is the idea that not every request needs the same model, and coarse-grained routing at the session or query level is already running in production. Token-level routing goes further: inside a single response, some tokens are trivial to predict and some are hard. Reasoning transitions, rare names, format boundaries. Send the hard ones to the big model, let the small model carry the rest, and you push the cost-quality frontier out.

Serving that idea is where systems break. Existing stacks assume one LLM per process. Under token-level routing, the models fall out of step: the fast model produces its token and waits for the slow one, batches stall, and the throughput advantage evaporates. TokenRouter's answer is request-centric programming, model-centric execution. You write routing logic from the perspective of one request. The runtime starts a subserver for each LLM and dispatches requests asynchronously, with a delayed-batching scheduler whose hyperparameters come from a mathematical throughput model rather than hand tuning.

The results are the widest spread in this cluster: 2.01x to 64.15x higher decoding throughput than existing systems. The lower end is what you get just from not breaking the batch. The upper end appears when the routing policy sends most tokens to the small model and the big model lights up only for the hard ones. I've reasoned about token-level routing before and assumed the policy was the hard part. This paper's argument is the opposite: the policy is the easy part, and the serving layer was doing all the damage. The code is public at github.com/thu-nics/TokenRouter.

DLoop: draft more, verify less

Speculative decoding buys a bigger effective step size: a lightweight draft model proposes tokens, the target model verifies them in parallel, and the accepted prefix advances you several tokens per target pass. The hidden assumption is that one draft stage deserves exactly one verification.

DLoop questions that. With a capable draft model, the target frequently accepts everything the draft produced, and the verification pass was pure overhead. Adaptive draft lengths fix this for autoregressive draft models, but not for parallel ones, because parallel drafting needs target hidden states for tokens that haven't been verified yet. DLoop's move is to loop: keep drafting while the draft model stays confident, verify all accumulated tokens in one batched pass, and spend the savings on fewer target forward passes. Loop-aware training keeps the draft reliable in those extra stages by exposing it to its own hidden states for unverified tokens. The repository is naver-ai/DLoop; the code was under internal review at time of writing.

The gain is 5 to 41% extra wall-clock speedup on top of methods like EAGLE-3, DFlash, Domino, DSpark, and multi-token prediction modules, and it preserves lossless decoding. Practical translation: a speculative pipeline that took 10 seconds per response drops to about 7 in the best case, for no accuracy cost, by spending a few more cheap draft forward passes.

Key numbers1.48x end-to-end decoding speedup on A100, SparseDecoding vs dense baseline. 2.01-64.15x decoding throughput, TokenRouter vs single-LLM serving systems. 5-41% added wall-clock speedup, DLoop over existing speculative decoding. ~70% fewer stored parameters, reViT-B/16 at DeiT III accuracy.

reViT: one block, many depths

reViT is the outlier: it's a vision paper, and it attacks the training side rather than the serving side. But the mechanism is exactly what this cluster cares about. A single transformer block, applied recurrently, matches the accuracy of a full-depth vision encoder at comparable inference FLOPs, without intermediate feature distillation.

The trick is a normalized-depth coordinate that reads the block's position in the network. The FFN at each depth becomes a convex combination of a small shared expert bank, and that coordinate programs the mixture. You get depth-specific behavior without depth-specific weights. In controlled comparisons at a matching one-FFN budget, weight-space merging (blending expert weights) beat token dispatch and output mixing, which is a useful data point for anyone betting on MoE structures.

The serving-relevant numbers: reViT-B/16 hits DeiT III accuracy while storing about 70% fewer parameters, so roughly a third of the original parameter bytes to load, which matters for checkpoint transfer, cold starts, and fitting weights alongside a growing KV cache. An 8-expert version distilled from a DINOv2 teacher keeps nearly all of the teacher's linear-probe accuracy and transfers to segmentation and depth prediction. Elastic-depth training means one checkpoint can run at multiple depths by resampling the coordinate interval. And for fixed-depth deployment, you can materialize the recurrent block into a conventional dense graph: no online routing or merging at runtime, same one-FFN-per-depth compute, at the cost of expanding deployed storage.

The transferable idea for LLM serving is the decoupling of stored parameters from used parameters. A stack that loads one block plus a small expert bank, and programs depth by a scalar, is a different memory equation than loading 32 distinct layers.

The five attacks side by side

Paper	Bottleneck	Mechanism	Headline result	Cost you pay
SparseDecoding	Weights read per token	Decoding-aware N:M pruning + SpMV kernel	1.48x end-to-end decode speedup on A100	Accuracy loss from pruning; needs N:M-capable kernel
VFold	Value cache memory	Symmetry-aware merging across layers	Compression neither quantization nor key pruning reaches alone	Small per-step merging overhead
TokenRouter	Model cost per token	Token-level routing, subservers, delayed batching	2.01-64.15x decoding throughput	New serving architecture; routing policy work
DLoop	Verification passes	Looped drafting before batched verification	5-41% extra speedup on top of spec decoding	Extra draft forward passes; loop-aware training
reViT	Stored parameters	Depth-programmed shared expert bank	~70% fewer stored params at DeiT III accuracy	Vision-first; needs training-time adoption

What trips people up

Five mistakes, one for each paper's lesson.

Pruning on stale text. If your calibration set is natural text and your model generates its own tokens, the Hessian you prune with doesn't match the activations you serve. I've seen a 2:4 pruned model look fine on calibration loss and visibly drift on long generation. Collect the calibration from the dense model's own generation, prefill excluded.

Compressing the KV cache layer by layer. Per-layer compression leaves cross-layer redundancy untouched, and that redundancy is exactly the capacity VFold exploits. Layer-wise thinking treats the cache as a stack of independent arrays when the values across layers are close cousins.

Running token-level routing on a single-LLM server. The routing policy is the fun part, but the serving layer is where token-level routing dies: step desynchronization and batch admission delays will eat your throughput. You need subservers per model and a scheduler built for async dispatch.

Verifying every draft stage immediately. When draft acceptance is high, you're burning target forward passes on tokens that were essentially certain. And the looped structure requires loop-aware training; a draft model trained to predict one stage ahead will misbehave when asked to draft several stages ahead on its own hidden states.

Treating these methods as alternatives instead of a stack. VFold composes with quantization and key pruning. DLoop sits on top of EAGLE-3. SparseDecoding prunes weights, VFold shrinks values, TokenRouter changes model choice. Different resources, mostly additive.

One thing to remember

Memory traffic per token is the binding constraint on decode latency. Every trick in this cluster either reduces the bytes read per token (SparseDecoding, VFold, reViT) or reduces the number of passes (DLoop, TokenRouter). When you evaluate one of these methods, measure throughput and latency at a target accuracy, on a long-context workload, and remember that several of them stack with what you already run.

What this means for your serving stack

If you're memory-bound on long contexts, start with VFold-style value cache merging. It needs no architecture change, it composes with the quantization and key pruning you already run, and it directly frees the memory that's capping your batch size.

If you're latency-bound on decode and can accept a small accuracy trade, SparseDecoding's decoding-aware N:M pruning is the right baseline, but only if your stack has an SpMV kernel that can actually exploit it. A 2:4 sparse model on SpMM-oriented kernels won't deliver the measured speedup.

If you run a fleet of models at different price points, token-level routing with TokenRouter is the biggest single lever here, at the cost of adopting a new serving architecture. Keep DLoop in your back pocket: it's lossless, it improves any speculative decoding pipeline you already run, and it costs only a few extra draft forward passes.