Five papers landed on the same underlying argument: the cheap way to make inference faster is to stop treating the model as a fixed object. Width, bit-width, where the weights live, and how long context keeps costing you are all becoming knobs you turn after training.

None of them require a new training run from scratch. None of them are the usual "we compressed the model and lost 3 points" story either.

The bill has four line items

You pay for inference in four places, and each has its own scaling curve.

Compute is what each token costs in FLOPs. Precision is how many bits each weight and activation occupies. Weight memory decides whether the model fits on hardware you already own. Context is the recurring bill for tokens the model has already read once.

The standard answers are pruning, post-training quantization, offloading, and retrieval. None of those are wrong. What changed is the granularity. Instead of one checkpoint at one size and one bit-width, the recent work makes these decisions per neuron, per expert, per request, or per corpus. That shift is the thread connecting all five papers.

Each one picks a different line item off the bill.

Elasticity: one checkpoint, a continuum of budgets

Your Transformer Is Secretly Elastic goes after compute. Model families ship in a handful of sizes, and you commit to one based on your latency budget, then live with it. Query load doesn't care about your choice. At 3am you're paying for width you don't need. At peak you're queueing behind your own batch jobs.

The paper's move is to rank every individual MLP neuron and attention head on a common scale, jointly, rather than assigning one importance weight per block or head like earlier work. They minimize loss under soft pruning across many sparsity targets at once, which turns sparsity selection into a differentiable unit-ranking problem instead of a separate search per budget.

One checkpoint then serves a continuum of compute budgets. At 50% sparsity, DINOv3 ViT-H+/16 loses 427M parameters and 0.8% ImageNet-1k linear-probe accuracy, with no retraining and no weight correction. At 60% sparsity, AugReg ViT-B/16 reaches 1.80x throughput and 1.26x lower single-query latency on an A100. If a throughput multiplier of 1.80x sounds abstract, it turns a ten-minute batch job into roughly five and a half minutes on the same hardware.

Key numbers from the elasticity paper

• 427M parameters removed: pruning DINOv3 ViT-H+/16 to 50% sparsity costs 0.8% ImageNet-1k linear-probe accuracy.
• 1.80x throughput and 1.26x lower single-query latency for AugReg ViT-B/16 at 60% sparsity on an A100.
• 27 encoders tested, spanning 22M to 8B parameters, with smooth degradation up to 60% sparsity.
• Elastic models are built in minutes, with construction time scaling sub-linearly with parameter count.

Scope matters here. Those 27 models are encoders, vision and text. The 22M end runs on a phone; 8B needs a real accelerator. Decoder-only LLMs with a KV cache have a different cost profile, and the paper doesn't claim otherwise.

What kept me from shipping per-request sparsity before was the eval matrix, not the accuracy. A pruned checkpoint gives you exactly one number to defend. A model that runs anywhere between full width and 60% sparsity means your eval suite has to sweep budgets, and someone has to decide what "good enough at 45%" means. That's a process problem, and it's usually the real blocker.

Quick Take: The interesting shift isn't any single compression trick, it's that compression is becoming a runtime decision instead of a checkpoint decision.

Mixed precision is an allocation problem now

Beyond Expert Rankings targets the precision line for MoE models. Sparse experts mean only a few activate per token, but all of them occupy storage. Mixed-precision quantization gives sensitive experts more bits and resilient ones fewer.

The gap the paper identifies is subtle and easy to miss. A ranking tells you the order of experts by sensitivity. It doesn't tell you how strongly to weight their errors. Two experts ranked first and second can differ enormously in how much damage a bad quantization does, and if you weight errors by rank alone you over-protect some experts and under-protect others.

NatScale calibrates gradient-based expert weights against an information-theoretic contrast target using a model-specific power transform. The ranking survives, the magnitudes get fixed. It evaluates the gate, up, and down projections of each expert together to estimate that expert's output error, then combines those errors with a prior built from weight spectra, variance, and precision decay to pick configurations under a storage budget.

Across three models and nine settings, it posts the lowest native-to-quantized KL divergence and the highest top-token agreement against five allocation baselines. It also lands the lowest perplexity in seven of those nine.

KL divergence is the metric that matters, not accuracy on a benchmark. It measures whether the whole next-token distribution shifted. If you sample with temperature, a slightly skewed distribution produces noticeably different text even when the argmax never changes. Top-token agreement is the blunt instrument; KL is the one that predicts whether your outputs drift.

Rotation choices without the calibration tax

SpectrumQuant sits in the same line item but attacks a different lever. Orthogonal transforms reshape activation distributions without changing what the model computes, which makes them a standard preprocessing step for low-bit post-training quantization.

Fixed Hadamard rotations are free and ignore your data. Learned rotations fit your data and cost hours of GPU time. SpectrumQuant works out rotation selection from first principles: maximizing covariance leads to spectral alignment, which gives a principled criterion for a quantization-friendly transform. That makes the rotation closed-form. Estimate the activation covariance from calibration data, build the transform, stop.

Hadamard matrices turn out to be a special case of their formulation, and a class of butterfly matrices falls inside the same family. On Llama and Mistral, SpectrumQuant lands in the same accuracy band as learned rotations while using less than one-third of the GPU memory and much less time.

If your quantization pipeline currently spends hours optimizing rotations on the same GPU you'd rather be serving from, this is the swap to make.

Rotation strategy	Uses your data	Calibration cost	Where it fits
Fixed Hadamard	No	Near zero	Fast fallback; a special case of the SpectrumQuant family
Butterfly matrices	Partly	Low	Structured and hardware-friendly, same family
Learned (SpinQuant, DartQuant)	Yes, optimized	Hours of GPU time	Best accuracy when you can afford the run
SpectrumQuant	Yes, one covariance estimate	Under one third of learned rotations	Data-aware, closed-form, cheap

Host memory is the cheapest capacity you own

COUNT takes the memory line from an unusual angle. N-gram embeddings scale memory separately from compute, because each forward pass reads only the embedding rows it needs. The problem is that training those embeddings from scratch burns GPU memory on fast random access. Meanwhile, the host system's ordinary DRAM sits mostly idle.

COUNT builds n-gram embeddings without adding learned parameters at all. It stores precomputed n-gram counts on the host and uses corpus frequencies to weight together the model's own unembeddings of the tokens that most often follow the current n-gram history. Nothing to train. Nothing to backpropagate.

At a fixed compute budget, adding COUNT as an extra input embedding during pretraining of a 1B model improves validation loss as much as an LM with 57% more compute. Read that as getting the loss curve of roughly a 1.57B model while paying for 1B. On a fixed cluster budget, that's the difference between a model that converges and one that runs out of allocation.

There's a second result that matters more for deployment. If GPU memory is limited, don't split it between backbone parameters and learned n-gram embeddings. Use COUNT and let the backbone fill the GPU. The comparison favors COUNT in that regime.

The third piece is training-free adaptation. Swap in a count table built from a domain-specific corpus at inference time and you shift the model toward that domain without a single gradient step. If you have a corpus but no labels, that's a much shorter path than a fine-tune.

Context that sticks without the prompt

Cortex goes after the fourth line item. Context improves output, and the moment you remove it, the improvement leaves with it. You keep paying to re-send the same context on every request.

Memory consolidation, as the paper frames it, internalizes that gain while preserving non-target capabilities. It's a common objective covering context-specified editing, skill learning, and behavioral steering, measured by a five-task-family testbed under single, batch, and sequential updates. Cortex generates prospective queries that anticipate how the context will be used, then distills context-conditioned outputs into mergeable low-rank or structural feed-forward network updates.

Across 4B to 120B dense and MoE backbones, text-only and multimodal, it sits on the consolidation-retention Pareto frontier in most evaluated settings. After 50 sequential factual edits on a 14B model, Cortex-Structural recovers 88.9% of the context-induced gain while retaining 99.8% of original non-target performance.

Both numbers matter and they point in different directions. The 88.9% says you don't get everything back. Roughly one ninth of the benefit still leaks away. The 99.8% says repeated edits barely erode what the model already knew, and that's the one that decides whether this ships. Catastrophic forgetting is why most teams refuse a long stream of sequential edits, and 99.8% retention after 50 of them is a different regime.

I've watched a team give up on incremental factual updates after a few hundred corrections, not because any single update failed, but because the aggregate drift became impossible to audit. A consolidation path that holds 99.8% retention is the first thing I've seen that makes that audit tractable.

Stacking them without breaking each other

These methods don't live in isolation, and the order you apply them changes the result.

Elastic pruning and quantization interact in a way that's easy to get wrong. Prune to 60% sparsity and the sensitivity profile of the surviving weights is not the profile you measured on the dense model. A bit allocation computed before pruning is stale. Decide width first, then allocate bits against the pruned network.

COUNT is orthogonal. It adds input embeddings and touches no backbone weights, so it composes with anything. Cortex is the awkward one. It writes FFN updates, which is the exact region elastic ranking cares about, so consolidating into a pruned model deserves a fresh ranking pass rather than a merge and a hope.

Method	Line item	Needs at build time	Retraining	Win shows up as
Elastic ranking	Compute	Minutes of construction per model	No	Per-request width selection
NatScale	Precision	Calibration batches, storage budget	No	Lower KL divergence, better top-token agreement
SpectrumQuant	Precision	Calibration data for one covariance estimate	No	Quantization at reduced calibration cost
COUNT	Memory	A corpus to count n-grams over	No	Loss equivalent to 57% more compute
Cortex	Context	A context stream, plus training of FFN deltas	Only the delta	Gains that survive context removal

What trips people up

Scoring whole blocks instead of individual units. Per-block importance is the baseline the elastic paper beats. If your pruning script still assigns one score per MLP block, you'll hit a cliff around 40% sparsity instead of the smooth curve. Move to per-neuron and per-head, and rank them jointly on one scale.

Weighting MoE expert errors by rank. Rankings give order, not magnitude. If you allocate bits using rank alone, noisy experts get under-quantized while the ones that actually mattered get squeezed. That's the exact gap NatScale was built to close.

Spending VRAM on n-gram tables you could keep on the host. Learned n-gram embeddings consume the GPU memory your backbone wants. When memory is the binding constraint, the COUNT comparison favors giving the whole GPU to the backbone and reading counts from DRAM.

Assuming every orthogonal transform is function-preserving. Hadamard and butterfly matrices are. A rotation you learned from calibration data is only as good as that calibration set. Check that outputs match on held-out inputs before you trust it.

Reading a 60% sparsity result as a general promise. The elastic paper validates up to 60% on encoders. Nothing in it promises a decoder LLM at 70%, and the failure mode is a slow accuracy bleed, not a crash you'll notice in staging.

One thing to remember: every method here moves a decision that used to be frozen in a checkpoint into something you can change after training. That only pays off if your serving stack can actually make the decision per request. A sparsity level or bit allocation you compute once and hard-code has exactly the rigidity you were trying to escape.

The Bottom Line

• If your traffic swings between idle and peak, elastify a single checkpoint instead of hosting two model sizes. Rank per neuron and per head, pick width per request, and treat the 60% sparsity result as your floor rather than your default. Two checkpoints means two weight sets resident in memory and two deploys to keep in sync.
• If you're cramming a large MoE onto fixed hardware, spend your engineering time on per-expert bit allocation before you touch rotations. Rankings are the easy part. The magnitude calibration is where the KL divergence drops, and SpectrumQuant gives you a closed-form rotation as the cheaper second win.
• Watch the Cortex line closely. Context consolidation that holds 99.8% of non-target performance after 50 sequential edits changes the economics of long-lived assistants. If a follow-up holds that retention at thousands of edits, retrieval stops being the default answer for personalization work.

References

• Your Transformer Is Secretly Elastic
• Beyond Expert Rankings: Contrast Calibration for Mixed-Precision MoE Quantization
• The COUNT: Counting N-grams Unlocks Scaling on Host Memory and Training-Free Adaptation
• SpectrumQuant: Spectrum Alignment for LLM Quantization
• Cortex: Consolidating Context into Parametric Memory