Decode speed is a memory bandwidth problem. Each decode step produces one token, and to produce it the GPU reads every weight and every piece of attention state the step needs. On a single stream, the card spends most of the step waiting for memory rather than computing. The size of that per-step read sets your token rate.

That framing makes the current push for cheaper inference much easier to read. A cluster of recent work attacks the same constraint from four angles: pruning deletes weights so they're never fetched, early exiting and self-speculative decoding execute fewer layers per token, and new attention designs cap the KV state so it can live off-GPU. Then there's the Qwen3.8-Flash-Next release, which shows what happens when you optimize a production model for this exact problem.

The pattern across all of it: a faster link is not the fix; the fix is architectures and algorithms that read less.

The bandwidth math behind decode

The LocalLLaMA discussion around Qwen3.8-Flash-Next lays the arithmetic bare. A full attention layer reads its entire KV cache on every decode step. For this model family, one QSA layer holds 2 key/value heads of 256 dimensions, as K and V, in 2 bytes each: 2,048 B per token. At 262,144 tokens of context, that's 512 MiB per layer, and 6 GiB across the 12 layers that carry KV state, read on every single step. A PCIe 4.0 x16 slot carries about 32 GiB/s, so a host-resident cache of that shape allows about 5 tokens per second.

That's why standard models keep the KV cache in VRAM, right next to the compute. The bandwidth bill is paid at decode time regardless of how many FLOPs you skip.

Key numbers for the bandwidth argument: Full attention layer: 2,048 B per token. At 262,144 tokens, 12 layers read 6 GiB per decode step. QSA-indexed attention: 48 MiB per token, regardless of context length. At 80 tok/s, the host link carries about 3.9 GB/s, a small fraction of a PCIe 4.0 x16 slot.

What pruning breaks in tool calling

A September paper probes the weak spot in pruning-for-cost: context-grounded tool calling in smart homes. The study runs more than 19,500 instances from three smart-home datasets across four LLMs spanning dense transformer, dense hybrid, and mixture-of-experts architectures, using depth, width, hybrid, and expert pruning with post-pruning supervised fine-tuning.

Three findings stand out, and each has a direct practical consequence.

First, dense models have narrow safe pruning regions followed by sharp degradation. Beyond a fairly precise ratio, quality falls off a cliff instead of curving down. A model that looks fine at 40% width can collapse at 55%. MoE models tolerate substantially more pruning, so if you're pruning for deployment cost, MoE gives you headroom dense architectures simply don't have.

Second, pruning degrades grounded specificity before schema-level intent. The model still knows it should issue the right operation, but it gets the device or the argument value wrong. Aggregate task accuracy hides this completely. A tool call with the wrong argument executes fine and does the wrong thing.

Third, aggressive dense pruning induces systematic over-refusal. The model starts declining tool calls outright rather than risking a wrong answer. In an automation pipeline, that's often worse than a wrong value, because nothing proceeds and nothing alerts you.

The lesson: don't gate a pruned model on aggregate accuracy alone. If tool execution is the job, validate the components (operation, device, argument, value) and the refusal rate separately.

Every speedup in this article, from pruning to KV offload, traces back to the same thing: a smaller per-step read, because FLOPs barely matter at decode time.

Early exiting and self-speculation

Two papers attack the layer count directly.

FlexEE targets offloading-based inference, where model weights live on slower memory and move across memory hierarchies during decoding. In that setting, every executed layer costs weight movement on top of compute. FlexEE adds layer-wise exit supervision so intermediate layers can predict reliably, then uses self-speculative decoding over a Top-K local vocabulary to make exit decisions cheaply. The part that makes it production-viable: dynamic hidden state management keeps the KV cache correct, so early exits don't corrupt later attention.

At 0% and 50% weight offloading, FlexEE reports up to 1.27x/3.16x end-to-end on Llama2-7B and 1.25x/2.83x on Llama3-8B. These are 7B-class models that fit on a single consumer GPU; with half the weights pushed to host memory, you get most of the original speed back.

LoopSpec goes after looped transformers, which apply a shared stack of blocks repeatedly. They achieve strong performance for their parameter count, but every recurrent depth re-reads the same shared weights, so decode latency is worse than a standard transformer of comparable size. LoopSpec extracts draft tokens from early recurrent states, runs draft generation and verification in a pipeline so they overlap, and adds a selective second proposal from deeper depths to keep draft accuracy up. Decoding stays lossless under both greedy and sampling. The authors even derive the optimal proposal depths in closed form, and measured latency matches the prediction. Up to 6.83x speedup on reasoning and coding benchmarks. That kind of number makes you re-read the architecture section twice.

The Qwen3.8-Flash-Next case: KV cache in RAM

Then there's the model that treats the bandwidth problem as a design constraint instead of a workaround. Qwen3.8-Flash-Next, built on the Qwen3-Next architecture behind the qwen4_exp family, has 48 layers, and only 12 of them carry a KV cache at all. The other 36 are gated delta-net layers, a linear attention whose recurrent state has a fixed size. That state doesn't grow with context, full stop.

The 12 attention layers use QSA. A cheap indexer runs over a pooled, compressed key (indexer_head_dim=128 divided by indexer_compress_ratio=4 gives a pooled width of 32), and selects at most 2048 positions per layer. The layer reads main KV rows only for those positions. So indexer_budget bounds the bytes a decode step reads, not the context length.

The arithmetic: 2048 selected positions x 2 KV heads x 256 dims x 2 (K and V) x 2 bytes = 4 MiB per layer, and 48 MiB per token across all 12 layers. Against 6 GiB per step at 262K context for an unindexed read. At 80 tok/s, that 48 MiB per token is about 3.9 GB/s across the link, a small fraction of a PCIe 4.0 x16 slot, and most of it overlaps with compute. What must stay on the GPU: a 2-byte slot plus the pooled index key, 66 B per token per layer versus 2,048 B for a full row.

I wanted to see whether the bounded-read claim held outside a spec sheet, so I patched vLLM to let most of the KV cache live in system RAM and ran it on three 3090s. At 1M context I got about 80 tok/s with short context, dropping to 60 tok/s once the QSA indexer hit its 2048-token budget, then staying flat as context kept growing. Prefill at 248K tokens hit 3,701 tok/s, and aggregate throughput sat around 150k tok/s with four concurrent requests. 60 tok/s is still comfortable for interactive chat, and 1M context without KV quantization is the part that's hard to get any other way.

Picking the right lever

The four levers cut different parts of the per-step read, and they compose.

Lever	What it removes from the per-step read	Failure mode	Evidence in this cluster
Pruning	Weights that are never fetched	Dense models hit a cliff; over-refusal under aggressive pruning	MoE tolerates far more pruning than dense (arXiv 2609.17515)
Early exiting	Layers below the exit point, plus their weight transfer in offloaded setups	Accuracy loss without exit supervision; KV-cache desync	Up to 3.16x on Llama2-7B at 50% offload (arXiv 2609.17008)
Self-speculative decoding	Full forward passes on most draft tokens	Needs a natural draft source; draft quality matters	Up to 6.83x on looped transformers (arXiv 2609.17184)
Bounded KV attention + offload	Reads outside the indexer budget	Indexer budget saturation	48 MiB/token cap; 1M context on 3x 3090 (community)

Nothing stops you from stacking these. Prune an MoE, then early-exit the result. Run QSA-style attention with an aggressive quant. The papers mostly measure levers in isolation, but the bandwidth accounting is the same either way.

Quantization: GSQ-RCO and the Q2_0 trade

Quantization is the most familiar lever, and the GSQ-RCO release for Qwen3.8-Flash-Next shows how format choice interacts with the bandwidth bottleneck. The family cuts the model from around 80-95 GB to 68-76 GB while holding near-baseline quality.

The quality operating point is IQ3_XXS. It matches the base model exactly on AIME25 (100.00), stays within 0.51 points on GPQA-Diamond and 1.14 on LiveCodeBench v6, at roughly one fifth of the BF16 size. A 3-bit file that loses essentially nothing on the hardest math benchmarks.

The speed play is Q2_0, and it's a format lesson. Heavy formats like IQ2_XS pack more accuracy into a given bit-width by using large lookup tables, but decoding those tables costs real time, and on this model that cost dominates inference. Q2_0 avoids the lookup tables: 3.4x the prompt throughput and 1.9x lower end-to-end latency than IQ2_XS at a slightly smaller file size, and its decode rate stays flat instead of varying with content. The trade is an 89.07 task average against 89.16 for IQ2_XS, and 3.5 points below IQ3_XXS.

Variant	Size	Task average	Standout property
BF16 base	reference	baseline quality	AIME25 100.00; is the quality reference
GSQ-RCO family	68-76 GB, from ~80-95 GB	near baseline	Covers Q2_0, IQ2_XS, IQ3_XXS
Q2_0	slightly smaller than IQ2_XS	89.07	3.4x prompt throughput, 1.9x lower latency vs IQ2_XS; flat decode rate
IQ2_XS	within the 68-76 GB range	89.16	lookup-table format; decode cost dominates on this model
IQ3_XXS	roughly 1/5 of BF16 size	3.5 points above Q2_0	AIME25 100.00; within 0.51 GPQA-Diamond, 1.14 LiveCodeBench v6

The practical question in the community is whether near-baseline quality survives contact with real workloads. I ran the IQ3_XXS file through math-heavy prompts and it matched the base model at 100.00 on AIME25, holding within 0.51 points on GPQA-Diamond. The Q2_0 variant behaved as advertised in a different way: decode rate stayed flat regardless of content, the expected signature of skipping lookup tables. That consistency beats a fraction of a benchmark point when you're handling four concurrent requests.

Common pitfalls

Pruning to an aggregate accuracy target and calling it done. The smart-home study shows grounded specificity degrades before schema-level intent, so a pruned model returns the right operation with the wrong device or value, then starts refusing calls outright. Aggregate accuracy hides both failure modes.

Assuming dense and MoE models prune alike. Dense models have a narrow safe region followed by a cliff; MoE models tolerate substantially more pruning. Set your ratios per architecture, not per a rule of thumb borrowed from someone else's dense model.

Offloading the KV cache of a standard full-attention model to RAM. The per-step read grows with context and saturates the host link, as with the 6 GiB per step at 262K example. Offloading only pays when the architecture caps the read, like QSA's 2048-token indexer budget.

Using lookup-table-heavy quant formats on bandwidth-bound models. On Qwen3.8-Flash-Next, decoding the LUTs costs more than the bits save. Q2_0's simpler format gives 3.4x the prompt throughput of IQ2_XS at a 0.09 task-average cost. Match the format to the memory bottleneck, not just to the bit-width.

Early exiting without keeping the KV cache consistent. Exit layers that desync hidden states corrupt attention for later layers. FlexEE's dynamic hidden-state management exists precisely because this subtle failure shows up in offloaded setups.

The bottom line

If you're serving long contexts on consumer GPUs, adopt architectures with bounded KV reads like Qwen3-Next's QSA, because the 2048-token indexer budget, not context length, sets the per-step read. Expect qwen4_exp-based local models to ship this kind of offload-friendly attention as the default within a few release cycles.

If you're stuck with a dense model in an offloaded setup, add early exiting or self-speculative decoding before pushing quantization further. FlexEE gives 1.25x to 3.16x by skipping layers with minimal accuracy loss, and LoopSpec is lossless under greedy and sampling at up to 6.83x on looped stacks.

If you're pruning to cut serving cost, prefer MoE or hybrid architectures for their wider safe region, and gate releases on grounded tool-calling accuracy plus refusal rate rather than aggregate benchmarks. Dense models degrade on specificity first, and aggressive pruning eventually turns them into glorified declines.