The bill arrives at prefill time

A twelve-image request doesn't stall in the decode loop. It stalls at prefill. Every image expands into hundreds or thousands of visual tokens, the model attends over all of them in one pass, and the KV cache entries for those tokens stick around until the request finishes. You end up paying for tokens the task never needed.

Four papers from the same week attack four different points in that pipeline. mmVB cuts visual tokens before prefill (arXiv:2610.18858). MultiAlloc decides how to spend a fixed KV cache memory budget (arXiv:2610.18642). SP²ec tunes the speculative draft during decode (arXiv:2610.18735). SCRouter decides which model should see the request at all, and with what output budget (arXiv:2610.18566).

None of them require retraining a 7B backbone. That's the common thread, and it's why they stack.

One pipeline, four places to cut

Order matters. Routing is first because it's the cheapest decision to make and the most expensive one to get wrong. Visual budgeting runs before prefill because that's where tokens are created. KV precision allocation runs as the cache fills. Speculative decoding runs last, once you're emitting tokens.

Optimize in the wrong order and you'll spend a sprint micro-tuning the decode loop while most of your wall clock sits in prefill waiting for image tokens to encode.

Make each image pay rent

mmVB is a plug-and-play module that assigns a query-conditioned token budget to each image using cross-image context. The base model stays frozen. Only 1.6M parameters get trained, which is roughly 3MB at fp16, noise next to a 7B backbone.

The design constraint that matters most is what it refuses to do. Many token-pruning methods read attention matrices to score token importance. That works fine in eager attention and falls apart everywhere else, because memory-efficient kernels like FlashAttention never materialize the attention matrix. Teams I've talked to hit this and either silently fall back to eager attention, losing the speedup they were chasing, or abandon the pruning method entirely. mmVB works without attention access, so it doesn't fight your serving stack.

Single-image pruning logic also doesn't transfer. Within one image, redundancy is spatial and local, so you can drop patches with little risk. Across images, the evidence for "which of these two products is cheaper" might be one label in image 3 and one number in image 11. Compress each image at a fixed ratio independently and you delete exactly those details. Budgeting across images with cross-image context keeps them.

The numbers translate cleanly. A 9x reduction in visual tokens means twelve images at roughly 1,200 tokens each drop from about 14,400 tokens to about 1,600, which is the difference between a 32k context feeling tight and feeling roomy. A 5x faster time to first token turns an 8-second wait into 1.6 seconds, which is the difference between a demo and something a user will tolerate.

That 94.4% figure is the one to interrogate. It means you give up roughly 5 to 6 points of full-resolution accuracy on average across eight benchmarks, and averages hide the tasks where the loss lands. Counting, OCR, and fine-grained comparison are the likely casualties.

Quick take: the cheapest inference wins here come from allocating scarce budget per query, not from making any single model faster.

Speculative decoding that retunes itself

Self-speculative decoding reuses part of the target VLM as the drafter, which preserves multimodal capability without training a separate small model. The catch is configuration. How deep should the draft go? How many layers should it reuse? The answer depends on the prompt, and a text-heavy prompt and an image-heavy prompt want different settings.

SP²ec runs a training-free bandit that adapts the draft configuration during generation. The design leans on a single-peak throughput pattern: for a given prompt, throughput as a function of draft depth rises to a maximum and then falls, and the peak sits in a different place depending on modality mix. The algorithm minimizes wall-time regret per query, which is the empirical latency you'd get if you'd picked the best configuration up front.

Up to 11% improvement over SWIFT, KnapSpec, SpecVLM, and a cosine-similarity baseline, measured on Qwen2.5-VL-7B-Instruct and LLaVA-NeXT-Mistral-7B. At 40 tokens per second, that's roughly 44. Modest, but it costs nothing in training and it composes with everything else on this list.

Method	Stage it touches	Training required	Reported gain	Where it stops helping
mmVB	Visual encoding before prefill	1.6M params, backbone frozen	9x fewer visual tokens, 5x faster TTFT, 94.4% accuracy retained	Text-only workloads with no images to budget
MultiAlloc	KV cache at prefill and decode	Offline profiling pass	Beats layer-wise, token-wise, and multi-granularity baselines at matched memory	Does not reduce prefill FLOPs
SP²ec	Decode loop	None, training-free bandit	Up to 11% over existing self-speculative methods	Helps least when draft and target agree on almost every token
SCRouter	Request routing and output budget	Probe plus utility/security predictors	13 to 29 points in-domain, 19 to 23 out-of-domain	Needs a labeled security set for your domain

KV cache precision is an allocation problem

Long context and reasoning workloads live or die on the KV cache, and quantization is the usual lever. The framing problem in MultiAlloc is sharper than "how much can I compress." Given a fixed memory budget, you have two ways to spend precision: raise the bit width on sensitive layers, or keep more tokens at full precision. You can rarely afford both without knowing where your accuracy actually comes from.

MultiAlloc profiles layer sensitivity and token importance offline, derives an allocation policy once, and applies it at inference with no online search. It beats layer-wise, token-wise, and multi-granularity quantization baselines at matched memory budgets across Mistral, Llama, DeepSeek-Qwen, and QwQ. The takeaway for anyone shipping a long-context product: a uniform 4-bit cache is leaving accuracy on the table at the same footprint. Judge these methods at a matched memory budget using accuracy, and ignore the compression ratio headline.

Key numbers

• mmVB: 1.6M trainable parameters, 9x fewer visual tokens, 7x fewer prefill FLOPs, 94.4% of full-resolution accuracy.
• SP²ec: up to 11% throughput gain over SWIFT, KnapSpec, SpecVLM, and a cosine-similarity baseline.
• MultiAlloc: matches a fixed KV memory budget across Mistral, Llama, DeepSeek-Qwen, and QwQ.
• SCRouter: 38,869 benchmark queries, 9 security dimensions, 144 model-budget pairs, plus 2,753 out-of-domain queries.

Routing is a constrained optimization

Standard LLM routers rank model-and-budget pairs by a single quality score and pick the winner. That works for narrow workloads and breaks on mixed traffic. For an ordinary query, a good answer is correct and useful. For a security-sensitive query, a good answer refuses a harmful request or resists a malicious instruction. Those two objectives prefer different models and different budgets.

SCRouter estimates, for every model-budget pair, the probability that the response finishes within budget, then feeds that probability into separate utility and security predictors. A small probe decides how to weight the two per query. The final selection balances weighted utility and security against cost.

The output-budget half deserves more attention than it gets. Granting a 4k-token budget to a model that would have finished in 200 tokens wastes money on every request. Truncating a reasoning model mid-chain guarantees a wrong answer that looks like a model failure. Budget is a first-class variable, not a rounding error.

SCRouterBench is 38,869 utility and security queries across nine security dimensions, each answered under all 144 combinations of 12 models and 12 budgets, plus 2,753 out-of-domain queries. Gains of 13 to 29 points in-domain and 19 to 23 points out-of-domain. That out-of-domain jump is the interesting part, because it suggests single-metric routers fail hardest on traffic they weren't tuned for, which is most production traffic.

What broke when I pushed these onto real hardware

I wired visual budgeting into a small multi-image pipeline, the kind of thing that compares product photos or reads a scanned contract. The token reduction held up on comparison prompts. It hurt on documents. Tables and small-print fields lost digits before anything else. That's what a 9x average reduction means in practice: it's an average across images, not a guarantee per image, and you need per-task accuracy checks before you flip it on for a document workflow.

The adaptive speculative tuning was the bigger surprise. I'd assumed a text prompt and an image-heavy prompt hitting the same model would want the same draft depth. They don't. Text prompts peaked at a deeper draft, image prompts peaked shallower, because visual tokens make verification more expensive per drafted token. A fixed config gave me roughly the average of the two peaks and never the peak, which is exactly the failure mode a bandit is built to avoid.

KV allocation is the piece I'd schedule first on a real team. Profiling is an afternoon of offline work, the inference path afterward is unchanged, and the win is accuracy at the same memory rather than a new deployment shape. Routers are the piece I'd schedule last, because out-of-domain behavior is where they get ugly and you need traffic samples before you trust the probe.

What trips people up

• Reading attention matrices for token importance while running FlashAttention. Memory-efficient kernels never materialize that matrix, so methods that need it either fail or quietly drop you back to eager attention. Your pruning then costs more than it saves.
• Reusing single-image compression ratios on multi-image prompts. Within one image, redundancy is spatial and local. Across images, the evidence for a comparison question can be one label in image 3 and one number in image 11, and independent per-image compression deletes exactly those.
• Setting one KV cache bit width for the whole model. Layers don't degrade at the same rate. A uniform 4-bit cache passes a smoke test, then falls apart on long-context reasoning where the sensitive layers do the most work.
• Ranking model-budget pairs by a single quality score. A router tuned for answer quality will send jailbreak attempts to your most capable model, which is also the most persuadable one. Utility and refusal need separate predictors.
• Freezing the speculative draft config at request start. Throughput curves are single-peaked and the peak moves with the prompt. Tune once, and you'll be fast on the exact prompts you tuned for.

One thing to remember

All four methods share one design decision: allocate per query at runtime instead of per deployment. A fixed token count, a fixed bit width, a fixed draft depth, and a fixed model choice are all the same mistake in different clothing, and each paper exists because that mistake is expensive.

The bottom line

If your traffic is multi-image, whether that's product comparison, scanned documents, or video frames, add visual budgeting before you buy bigger GPUs. A 9x cut in visual tokens carries a 5x cut in time to first token, and the 1.6M-parameter adapter means you keep your existing checkpoint and your existing serving stack.

If you're already quantizing the KV cache, stop tuning a single bit width. Profile layer sensitivity once, split the memory budget between layer precision and token preservation, and compare methods at matched memory instead of matched compression ratio. The MultiAlloc results across Mistral, Llama, DeepSeek-Qwen, and QwQ suggest this is free accuracy for a fixed memory footprint.

If your queries mix benign and adversarial traffic, add a security head to the router and treat the output budget as a tunable. Ranking by quality alone sends jailbreak attempts to the most capable model you have, which is the one you least want answering them. Gains of 19 to 23 points out-of-domain tell you where single-metric routing breaks first.