The bill comes from four places

Your serving cost is not one number. It's at least four, and they scale on different axes.

Prefill scales with sequence length. Parameter memory scales with the number of weights and, increasingly, with embedding tables that do no arithmetic but must be read anyway. Decode is serial, so any per-token overhead multiplies by the number of tokens you generate. And speculative decoding, the standard fix for serial decode, spends target-model compute on draft branches that mostly get rejected.

Four recent papers hit each of these. They don't compete. They stack.

That's the map. Now the details.

Sequence compression is a regression problem in disguise

Pooling and learned projections are the default tools for shrinking a sequence, and they both leave accuracy on the table. TTR-Compressor reframes the job.

Treat the input tokens as defining a regression problem. Fit it. The fitted coefficients are your compressed representation. The regression objective itself is learned end-to-end by backpropagation, but the coefficients come out of a closed-form solve during the forward pass. You get a learned notion of "what matters" without paying for an expensive learned encoder at inference time.

The paper tests it in two spots. As the compressor inside block-wise sparse attention, it beats attention-based compressors on RULER by a wide margin. As the visual token merger in a multimodal LLM, it matches the standard MLP merger while using 55.6% fewer visual tokens.

That second number is the one with a bill attached. On a 1,024-token image tile, 55.6% fewer tokens means roughly 570 tokens you don't push through prefill, and prefill is where multimodal models burn time-to-first-token.

Quick Take: The pattern across all four papers is the same: stop optimizing the intermediate representation and optimize what the intermediate representation has to accomplish downstream.

Per-layer embeddings are the new memory hog

Per-layer embeddings (PLE) give each transformer layer its own embedding table, so embedding capacity grows alongside compute. In Gemma 4 E4B, those tables account for 35.2% of all parameters. More than a third of the model sits in lookups that get read on every forward pass and contribute nothing to FLOPs.

Compress them naively and you hit a wall. At 100x compression, a conventional quantizer gets a few bytes per token-layer vector. That's not enough codebook to work with, and product quantization, the strongest of six baselines tested, keeps only 66.3% of the uncompressed model's aggregate benchmark score.

OPTIC takes a different angle. Store one quantized code tuple per token, then decode it at every layer with a layer-specific affine decoder. Codes and decoders are fitted to the PLE output path, not to the stored vectors. You aren't trying to reconstruct the embedding. You're trying to reconstruct what the embedding does to the residual stream.

PLE compression vs BF16	OPTIC retention	Product quantization
~100x	97.8%	66.3%
594x	90.5%	not reported
1,002x	84.1%	not reported

Retention is the aggregate across seven benchmarks. At 100x, OPTIC loses 2.2 points. At 1,002x, where each token's entire per-layer embedding budget is a rounding error, it still holds 84.1%.

The memory math is what sells it. At roughly 100x PLE compression on BF16 E4B, the model uses 29.7% less GPU memory at the same KV cache size while running at most 0.40% slower. Under half a percent is inside run-to-run noise on most clusters. If your model plus PLE footprint was 10GB, you just got about 3GB back to spend on batch size or context.

The result also holds on the official W4A16 release (4-bit weights, 16-bit activations): 96.9% retention at 100x. That matters because it means OPTIC composes with weight quantization instead of fighting it.

Key numbers:

• 35.2% of Gemma 4 E4B's parameters live in per-layer embedding tables
• 97.8% retention at ~100x PLE compression, against 66.3% for product quantization
• 29.7% less GPU memory at inference, at a cost of 0.40% throughput
• 84.1% retention even at 1,002x compression

Quantization-aware training that respects geometry

Lattice vector quantization gives good rate-distortion curves at low bit widths. What happens when you fine-tune through it has been mostly guesswork, and the standard approach is a single straight-through estimator applied to the whole quantizer.

SQuAT splits the quantizer into shape and gain, and gives each its own STE. Weight directions get normalized and quantized against an E8-derived spherical codebook. Magnitudes get quantized separately with low-bit scalar quantizers. The paper shows that this split produces tangent-space updates for the direction component, which is an implicit form of Riemannian optimization on the sphere. A single STE over the full quantizer ignores the shape-gain structure and loses that property.

The practical problem with lattice VQ at scale is nearest-neighbor assignment. Exact search over a large codebook, for every weight block, every step, is brutal. SQuAT distributes it: workers quantize disjoint weight blocks and communicate only discrete indices. Training time drops 12.8x, which is what makes a 10B-token run possible at all. A run that would have taken 12.8 days takes one, and that's the difference between an experiment you can afford and one you can't.

Tested across instruction-tuned LLaMA-3.2-1B and 3B, Qwen3-4B, and base Qwen3-1.7B, SQuAT beats standard VQ-QAT with the identical quantizer and initialization. Same codebook, same starting point, better accuracy. That's a clean ablation.

Speculative decoding wastes target compute on bad branches

Tree-based speculative decoding verifies several candidate branches in parallel. In production, similar requests keep arriving, so retrieval-based drafting reuses historical drafts and builds trees from them. The trees grow. So does the junk.

Every low-quality branch in that tree costs target-model forward passes for a token that will be rejected. ValuSpec filters the tree before verification starts.

The mechanism is deliberately boring in the best way. No parameter training. A filtering threshold is estimated from the target model's own greedy trajectories on a calibration split, then a transfer-based supervision signal aligns the filter with target behavior. Verification runs under the output-identical greedy rule, so accepted tokens match standard autoregressive decoding exactly.

Results: up to 2.38x end-to-end speedup in retrieval-based tree settings on HumanEval, GSM8K, Dolly, and SpecBench. And it drops into EAGLE-3's dynamic tree framework with no change to tree construction, reaching up to 2.78x. If your baseline single-stream decode was running 40 tokens/s, 2.78x puts you near 110.

Side by side

Technique	What it shrinks	Inference overhead	Reported gain	Where it fits
TTR-Compressor	Sequence length	Closed-form solve per forward pass	Beats attention compressors on RULER; matches MLP merger with 55.6% fewer visual tokens	Long-context prefill, multimodal token merging
OPTIC	Per-layer embedding tables	Layer-wise affine decode	97.8% retention at ~100x, 29.7% less GPU memory	Models with large PLE, memory-bound serving
SQuAT	Weight bit width	None beyond the quantizer	SOTA VQ-QAT accuracy, 12.8x faster QAT	Training-time compression for low-bit deployment
ValuSpec	Wasted verification passes	Small filter pass	2.38x, up to 2.78x with EAGLE-3	Any tree-based speculative decoder

Three of the four are inference-time. SQuAT is a training-time change with an inference-time payoff, which is why it stacks cleanly with OPTIC: SQuAT compresses the weights, OPTIC compresses the tables that quantization handles badly.

What I ran into

I keep a small eval harness that runs a handful of tasks and reports an aggregate, mostly to catch regressions when I swap compression schemes. The lesson that came out of it: reconstruction error is a liar.

When I first tried compressing a per-layer embedding table, I tuned my quantizer against mean squared error on the stored vectors. MSE went down nicely. The aggregate benchmark score went down with it. The vectors that matter are not the ones that are easy to reconstruct, they're the ones whose errors survive the residual stream and reach the logits. OPTIC's 97.8% vs 66.3% gap is the same lesson with a bigger codebook, and it's why the papers that fit to the output path beat the papers that fit to the vector.

On the decoding side, I learned to check batch size before trusting any speculative decoding number. Single-stream benchmarks are where these speedups live. Push concurrency up and the target model's verification batch saturates, so the headroom that made 2.78x possible gets eaten by queueing. If you serve at high batch, measure at high batch or you'll ship a number you can't reproduce.

The third thing: VQ-aware training is not a drop-in. If you take an existing QAT recipe and swap in a lattice quantizer, the hyperparameters you tuned for scalar quantization are wrong. The straight-through estimator needs to respect the quantizer's geometry, or you spend a week wondering why your loss curve looks fine and your eval doesn't.

Common pitfalls

Judging a compressor by reconstruction loss. MSE on the intermediate representation is the wrong objective when the representation feeds a downstream computation. Fit to the output path, evaluate on task metrics. Product quantization wins on vector error and loses by 31 points on benchmark retention.

Applying one STE to a structured quantizer. Shape and gain have different error geometry. Sharing a single straight-through estimator across both throws away the tangent-space update that makes SQuAT work, and you end up with standard VQ-QAT accuracy while paying lattice-VQ compute.

Quoting the memory saving as if it's the whole footprint. That 29.7% figure is measured at a fixed KV cache size. If your KV cache dominates the memory budget, the same PLE compression buys you proportionally less headroom. Do the arithmetic against your actual deployment, not against the paper's.

Trusting the exactness guarantee outside greedy decoding. ValuSpec's accepted tokens match standard autoregressive decoding exactly under the output-identical greedy rule. Serve at temperature 1.0 with top-p sampling and that guarantee no longer describes what you shipped. It doesn't break anything, but your correctness argument does.

Calibrating a filter threshold on the wrong distribution. The threshold for candidate valuation comes from the target model's greedy trajectories on a calibration split. Use a split that looks like production traffic. Calibrate on short prompts and deploy on long ones and the filter will cut branches it should have kept.

One thing to remember

Every result here comes from fitting the compressed object to the downstream computation rather than to the thing being stored. OPTIC fits codes to the PLE output path. TTR-Compressor learns the regression objective by backprop while solving coefficients in closed form. SQuAT's separate estimators produce updates that respect the sphere. ValuSpec calibrates its threshold from the target model's own behavior. The compression target is always the model's output, never the tensor.

The Bottom Line

If you serve a model with large per-layer embeddings and you're memory-bound, adopt OPTIC-style token-shared codes. At roughly 100x compression you keep 97.8% of accuracy and free up 29.7% of GPU memory at a 0.40% throughput cost. That's the best memory-per-point trade in this batch, and it survives W4A16.

If you run tree-based speculative decoding with retrieval-based drafting, add a valuation filter before verification. The 2.38x is on the table without touching your draft model, and 2.78x is available inside EAGLE-3 with no change to tree construction. Skipping it means you keep paying target compute for branches you already know are bad.

If you're training a low-bit model with lattice VQ, don't ship a single-STE recipe. Split the estimator across shape and gain, and budget for distributed exact nearest-neighbor assignment, which cuts QAT wall time 12.8x. One thing to watch: these four techniques are individually validated but largely untested in combination. Expect the next round of papers to report stacking results, and expect at least one interaction to be harder than the sum of its parts.

References

• Rethinking Sequence Compression with Test-time Regression: arxiv.org/abs/2610.31574v1
• Extreme Compression of Per-Layer Embeddings via Cross-Layer Latent Vector Quantization: arxiv.org/abs/2610.31309v1
• SQuAT: Spherical Lattice Vector Quantization-Aware Training: arxiv.org/abs/2610.31412v1
• ValuSpec: Plug-and-Play Candidate Valuation before Target Verification for Tree-Based Speculative Decoding: arxiv.org/abs/2610.31338v1