The stack is on-policy in your head, not on the wire

Setting up RLHF looks like plumbing. You load weights into vLLM for rollouts. You load the same weights into Megatron for training. You set BF16 everywhere. The weights are identical bit for bit. You assume the rollout distribution is exactly the policy being optimized.

It isn't. The two stacks do the same math in different orders, with different interim precisions, through different kernels. RoPE gets computed once and cached, then re-applied at attention time through a slightly different exponent path. SwiGLU's SiLU lands at BF16 a couple of multiplies earlier in one stack than the other. RMSNorm residuals drop to BF16 either before or after the normalization, depending on the fusion.

Each difference is small on its own. The problem is that they compound. Over a rollout of a few thousand tokens and a batch of a few thousand sequences, the policy you're training is measurably off from the policy you sampled. You think you're doing on-policy PPO. You're doing off-policy PPO with an unknown reference that drifts as the policy updates.

That's the first failure mode. The second is that rollouts are slow. The NgramRL paper puts the number plainly: autoregressive rollout generation can exceed 70% of training time. At that split, shaving rollout time is the highest-leverage optimization in the loop.

Three papers, one cluster. Two attack cost. One attacks stability. None of them touch the learning objective. That's the pattern worth studying.

Where rounding happens, and why it matters more than people think

The stability paper asks a narrow question: instead of aligning the whole inference and training stacks, can you align just the rounding locations that matter?

They pick three: RoPE, SwiGLU, and the residual RMSNorm. In each, they keep the intermediate computation in FP32 and only round to BF16 at the output boundary. That matches what the inference engine does. Everything else in the training stack, the matmul and attention kernels, the inference engine, the learning algorithm, stays as it was.

The default BF16 baseline collapses across four Qwen-family models. Aligning all three regions prevents the collapse. RoPE-path alignment alone also prevents it in the runs they studied, even though other numerical discrepancies remain.

The RoPE-only result is the one I'd test first in your own setup. You don't swap your inference engine. You don't swap your training stack. You keep FP32 intermediates in three specific places until they need to become BF16. That's a few lines in the model definition, not a stack replacement. If RoPE alone holds, it's a one-line fix.

Key Numbers

• 4 Qwen-family models tested
• 3 rounding regions aligned: RoPE, SwiGLU, residual RMSNorm
• 1 region sufficient on its own: the RoPE path
• 0 changes to the inference engine, matmul and attention kernels, or learning objective

Quick Take: the cheapest stability fix in LLM RL is to match where your training stack rounds to BF16, not to replace your whole inference stack.

KnitTrans: the pipelining bill

Separating rollout and training into different services is standard now. It's the only practical way to run them on different hardware with different parallelism. The problem is version skew. In a colocated setup, the trainer reads a trajectory drafted by the same parameter version it's about to update. In a disaggregated setup, batches of trajectories finish at different times and the learner pulls whatever is ready. The version gap between the model that produced a trajectory and the model currently training grows.

The KnitTrans paper calls this architectural staleness, and the definition pins it specifically to disaggregation rather than async gradients. That's a useful separation because it tells you the fix is in the pipeline, not the optimizer.

Their design has three parts. Constraint-preserving mini-batch admission: instead of waiting until a full train batch's worth of trajectories is ready, admit what's ready, as long as enough capacity remains to hit the remaining quotas. On-demand trajectory access: match storage layout to which micro-batch will consume the data, and overlap the data prep with the compute. Multimodal deduplication: if a mini-batch has the same image or audio in multiple samples, encode it once and reuse the encoding while preserving the gradient contribution from every backbone micro-batch that saw it.

The headline numbers: 93% lower architectural staleness than train-batch admission, and on multimodal workloads a two-order-of-magnitude reduction in transfer latency. Transfer plus training gets up to 5.8x faster than baseline.

The 93% figure does most of the work. It closes most of the quality gap between disaggregated and colocated execution. That's the difference between "we accept a small quality hit for a big throughput win" and "we get colocated quality at disaggregated scale."

NgramRL: the rollout bill

Speculative decoding in RL rollouts has a different shape than in inference serving. In serving, you have one model and you want to accelerate it for whoever's using it. In RL, the target model changes under you every training step. Any drafter you train or load onto the GPU drifts from the policy almost immediately, and it occupies memory the rollout engine needs.

NgramRL's observation is simple: recent rollouts of the same prompt are a high-quality draft source. The model has already generated continuations for this prompt, often across multiple training steps. Those continuations self-update as new ones replace old ones, and they're free to store if you cap the history.

They build a per-prompt n-gram drafter from the bounded recent rollout history. No auxiliary model. No drafter training. No persistent GPU-resident drafter state. The target model verifies, so the rollout distribution is preserved.

The problem with a CPU-side n-gram drafter is lookup latency eating the wins. They address it with an adaptive fan-out prefetcher that predicts the likely verification outcome, the pair (j, b) where j is the accepted draft length and b is the target-emitted continuation token, and prefetches on predicted outcomes. A causal budget allocator concentrates prefetches on high-probability outcomes. Anything uncovered falls back to sequential lookup without changing committed tokens.

Results across two model families and three workloads: 3.60 to 7.77 tokens committed per target-model forward in the warm-cache regime. That's up to 1.99x more than SuffixDecoding and 7.0x more than prompt-lookup decoding. On Spider, the prefetcher covers 96.6% of Qwen3-8B outcomes at a candidate budget of 16 and delivers 1.38x measured rollout throughput over synchronous NgramRL, close to the 1.40x overlap ceiling.

The 3.60 to 7.77 range is the number to keep in your head. Worst case, you commit about 3.6 tokens per forward, which is a 3.6x reduction in target-model calls per rollout. Best case it's closer to 8x. That's real headroom for a technique that adds no parameters and no GPU state.

Where each fix sits in the loop

All three interventions slot into the same rollout-to-training cycle. The diagram is the mental model.

NgramRL makes the slow loop faster. KnitTrans shortens the path from generated trajectory to trained-on trajectory. The rounding paper keeps the loop from diverging. They don't conflict with each other. A production run could reasonably apply all three.

Side-by-side

Paper	Bottleneck	What it changes	What it leaves alone	Headline result
Rounding-Location Alignment	Silent numerical drift between inference and training stacks	FP32 intermediates at RoPE, SwiGLU, residual RMSNorm until BF16 output	Inference engine, matmul and attention kernels, learning objective	Prevents collapse across 4 Qwen-family models
KnitTrans	Architectural staleness and trajectory transfer cost	Mini-batch admission, on-demand access, multimodal dedup	Rollout engine, training algorithm	93% lower staleness; 5.8x transfer+train speedup; 100x lower transfer latency
NgramRL	Autoregressive rollout cost	Per-prompt n-gram drafter from recent history, adaptive prefetcher	Target model, drafter training, GPU-resident drafter state	3.6-7.8 tokens per forward; 1.38x rollout throughput

The rightmost columns are the tell. All three leave the learning algorithm unchanged. None of them say "your reward model is wrong" or "swap PPO for GRPO." The fixes live in the infrastructure around the algorithm, not the algorithm itself. That's a mature pattern for a field where the objective is still being tinkered with.

The speedup headroom, visualized

Two of these are within-paper comparisons against a strong baseline (SuffixDecoding) and a weak one (prompt-lookup), so read the 7.0x accordingly. The 1.38x comes from the Spider workload and is the more honest number for what you'd get over a tuned baseline. The 5.8x bundles the transfer reduction with the throughput gain from lower staleness.

What trips people up

1. Assuming identical weights mean on-policy. They don't. The default BF16 baseline collapses in the rounding paper's setup with no algorithmic bug at all. Check where your training stack rounds to BF16 in RoPE, SwiGLU, and RMSNorm before you blame the optimizer.

2. Clearing rollout history after every weight update. NgramRL's drafter is the recent history of the same prompt. If you drop that cache to free memory, you've removed the draft source. Cap the history instead of purging it.

3. Admitting full train batches to keep the learner busy. That maximizes architectural staleness, which is exactly what KnitTrans is designed to avoid. A fraction of a second of learner idle time costs less than the quality hit from stale trajectories.

4. Reaching for a model-based drafter by default. In RL specifically, a trained drafter eats memory the rollout engine needs and drifts as the policy updates. The per-prompt n-gram approach avoids both problems at the cost of a lower ceiling.

5. Changing the learning algorithm when the run diverges. In the rounding paper's setup, the collapse looks algorithmic but isn't. Match the rounding locations first, then touch the optimizer.

One thing to remember

All three papers share a premise: when an RL run is slow or unstable, check the plumbing before you check the math. The plumbing is where the silent failures live. Rounding mismatches don't throw errors. Staleness doesn't show up in the loss curve until it does, and by then you've burned a week of GPU hours. A drafter that drifts just quietly stops