Appearance
Context Packets, Token Gates, and Routers: Three Levers on LLM Serving Cost
Prefill is the line item nobody budgeted for
If you run a production LLM system, you already know where the invoice goes. Decode gets all the attention because it's the part users watch stream in. Prefill is the part that quietly costs you 60% of the bill on long-context traffic.
Worse, prefill is repeated. Ask five questions about the same 80-page contract and the model chews through those 80 pages five times. Hand the same context to a second model in a pipeline and it does it again from scratch.
Three papers landed this cycle that each cut a different piece of this. They don't compete, and confusing them for alternatives is the most common mistake I've seen in internal design discussions.
| System | Acts on | Mechanism | Headline result |
|---|---|---|---|
| LatComm | context transfer between models | frozen encoder writes a fixed-size packet; receiver cross-attends with gates derived from its own prompt | +10.2 pts accuracy over single-vector baselines across 12 QA datasets; 57x smaller receiver input on QuALITY |
| TokenGate | prefill inside a single model | token importance scores re-weight attention logits, making top-K pruning differentiable | 1.4x to 11.3x faster TTFT, 1.5x to 3.4x lower peak memory, no accuracy loss |
| RouterWorld | model selection | benchmark with partial observability and hardware-grounded latency labels | 371k single-turn responses, 76k agentic trajectories, 13 baselines evaluated |
Different layers, different failure modes, different amounts of engineering. Let me walk through each, then show where they combine.
LatComm moves the context out of the token stream
The problem LatComm solves is specific: you have one context and many questions, and the context needs to reach a model that didn't encode it.
Text passing is readable and reusable but not compact. Single-vector compression is compact but throws away detail and isn't really designed for a second model to read. LatComm's claim is that you can have all three, and the way it gets there is by separating encoding from reading.
The encoder is a frozen LLM. It sees the context and nothing else, no question, no task hint. It writes a fixed-size packet. A different frozen receiver LLM then reads that packet through cross-attention, with the gates computed from its own prompt.
The encoder never sees the question. That single design decision is what makes the packet reusable, and reuse is what makes the economics work. Encode once, then answer two hundred questions against the cached packet.
The number that matters most here isn't the accuracy gain. It's the 57x reduction in receiver input on QuALITY. A 57,000-token article becomes roughly 1,000 tokens of packet, which means the receiver's prefill drops by nearly two orders of magnitude. Answering an article's full question set runs 3x faster end to end. If your product is "chat with this document," that's the difference between a 4-second wait and something that feels instant.
The 10.2 point average gain over single-vector baselines is the piece I'd flag to anyone already doing embedding-based context compression. Averaged over 12 multiple-choice QA datasets, a 10-point jump is not a tuning artifact. Single-vector compression loses information at the bottleneck, and giving the receiver a gate it can steer per question recovers most of it.
TokenGate teaches the model to skim
LatComm assumes the context is shared. TokenGate assumes it isn't.
Human readers don't process a codebase uniformly. They skim, then read closely what matters. Transformers push every token through every layer, which is why a 200K-token input costs what it costs.
The obvious fix is top-K token pruning, and the obvious problem is that top-K isn't differentiable. You can train a scorer with a proxy loss, or train it separately, or hand-tune it. None of those give you an end-to-end signal.
TokenGate's move is token-gated attention: instead of hard-selecting tokens during training, it re-weights attention logits with learned token importance scores. The mechanism borrows from mixture-of-experts routing. Soft gates during training keep gradients flowing, and you can hard-prune at inference when you actually need the speed.
The reported ranges: 1.4x to 11.3x faster time-to-first-token and 1.5x to 3.4x lower peak memory, with no accuracy degradation. Against the best prior pruning methods specifically, 6.4x faster prefill and 2x lower memory.
Translate those into your own numbers. A 100K-token prefill that currently takes 4 seconds to first token lands around 350ms at the 11.3x end of the range. Peak memory at 40GB of activations drops toward 12GB at 3.4x, which is often the difference between fitting a second concurrent request on the same GPU and not.
One architectural detail worth noting: TokenGate extends both standard transformers like Qwen3 and hybrid-attention models like Qwen3.5, and the adaptation is described as comparable in cost to standard parameter-efficient fine-tuning. That matters because hybrid-attention architectures are where a lot of long-context serving is heading, and pruning methods that only work on vanilla attention quietly stop applying.
Quick Take: These three systems cut different costs, so the fastest way to waste a quarter is to evaluate them head to head instead of in sequence.
RouterWorld measures routing when you can't see the future
Every routing benchmark I've used before this one shares a flaw: it assumes you know what every candidate model would have answered. In production you never do. You call one model, you get one answer, and the counters you didn't call are dark.
RouterWorld is built around that constraint. It bundles 371k single-turn responses and 76k multi-turn agentic trajectories, uses natural user interactions scored by a standardized LLM-as-a-judge protocol, and grounds latency in hardware measurements rather than provider price sheets.
The partial observability split is the part I care about:
- Cold-Start restricts learning to sparse, static logs, which is what you actually have on day one.
- Voyager models a continuous stream where routing decisions shape the future training distribution.
Voyager is the one that should worry you. A router trained on your production logs learns from a distribution its own decisions created. If it always sends a query class to the cheap model, you never observe the expensive model's outcomes for those queries, and the router's confidence in its own policy becomes self-confirming.
Across 13 routing baselines, the finding is blunt: most routers over-emphasize accuracy and ignore latency when observability is partial. That's a benchmark artifact meeting a production requirement. Accuracy is easy to score offline. Latency under partial observability requires hardware labels most benchmarks never collect.
Key numbers
- 371k single-turn responses in the RouterWorld corpus, enough to train a router on realistic query mix rather than synthetic prompts.
- 76k multi-turn agentic trajectories, which is where most routing policies break because early routing decisions constrain later turns.
- 13 routing baselines evaluated, with most underweighting latency.
- 57x reduction in receiver input from LatComm on QuALITY.
- 10.2 points average accuracy gain over single-vector compression baselines.
- 6.4x prefill speedup from TokenGate against the best prior pruning method.
A serving path that uses all three
The reason these stack cleanly is that they sit at different points in the request lifecycle and none of them requires the others to be present.
Order matters more than you'd expect. Cache lookup comes first because a packet hit eliminates the encoder call entirely, and the encoder is the expensive fixed cost. Routing comes second because model choice changes the prefill shape, which changes how much TokenGate can prune. Pruning comes last because it operates inside whichever receiver the router picked.
I've seen teams invert this, routing before checking the cache, and then pay for an encoder run on a document they'd already compressed an hour earlier.
What trips people up
Re-encoding per question. LatComm's entire win comes from the packet being question-independent. Call the encoder inside your request handler and you've built the slowest possible version of the system. Key the cache on a content hash and set an eviction policy before you ship.
Training TokenGate with hard top-K in the forward pass. The gradient dies at the selection operator. Use the soft gated attention during training and only hard-prune at inference. This is the same mistake people made with MoE implementations a few years back.
Calibrating a router on full-observability logs. If your training data contains outcomes for every candidate model on every query, your router learns a policy that can't survive contact with production. Validate on Cold-Start data or accept that your offline metrics are fiction.
Measuring latency from provider price sheets. Prices change, and a per-token price says nothing about time-to-first-token on a specific GPU with a specific batch size. RouterWorld's hardware-grounded annotation exists precisely because routers trained on the wrong latency signal optimize for a number that doesn't exist.
Assuming a pruning ratio is a constant. The 1.4x to 11.3x TokenGate range is wide because the achievable pruning depends heavily on task type. Benchmark on your own traffic mix before quoting a number to anyone who signs the GPU purchase order.
Where each one stops working
| Approach | Breaks when | What to do instead |
|---|---|---|
| LatComm | the question needs verbatim detail the fixed-size packet dropped, like exact identifiers, line numbers, or quoted clauses | keep a retrieval escape hatch that pulls exact spans for citation-heavy queries |
| TokenGate | you can't run the adaptation step on a model you serve, or you have no prefill-shaped training data | start from the released code and restrict pruning to layers where attention is already sparse |
| RouterWorld | you want a production router as a dependency | treat it as an evaluation harness for your own router, not a drop-in component |
The LatComm limitation is the one that caught me. Compression to a fixed-size packet is a lossy operation by construction, and the receiver's gates can only select what survived. For summarization and question answering that's fine. For "what's the value of the third parameter on line 412," you need the raw tokens.
Latency is the metric routers still get wrong
Routing benchmarks have spent years optimizing for answer quality because quality is cheap to measure and latency is not. RouterWorld's contribution isn't a new router, it's an argument that a router which picks the best model 90% of the time but blows the latency budget on the other 10% is worse than one that picks a good model every time and stays inside the budget. The 13 baselines it evaluates mostly fail that test. If you're building model orchestration today, instrument real time-to-first-token per candidate model on your own hardware before you train anything.
What this means for your stack
If your traffic is many questions against the same long documents, build the packet cache first. LatComm's 57x input reduction on the receiver side is the largest single lever in this group, and the encoder cost amortizes across every subsequent question, so the first hour of engineering buys you the biggest invoice change.
If your context is unique per request and there's nothing to reuse, skip the encoder and go straight to TokenGate. A 1.5x to 3.4x drop in peak prefill memory lets you raise concurrency on the same hardware, which usually beats a latency win in cost-per-query terms.
One thing to watch: latency annotations under partial observability are about to become the standard acceptance test for routing vendors. RouterWorld shipped the harness for it, and once a benchmark like this exists, procurement teams start asking why your router wasn't evaluated on it. Expect routers that report accuracy without a hardware-grounded latency number to look incomplete within a year.