Appearance
Post-Training Is Running Out of Data and Teachers. Six Papers Route Around Both.
The two inputs post-training depends on are both drying up
Every reasoning model you have used in the last two years got shaped after pretraining by two things: a pile of verifiable tasks and a teacher stronger than the student. Both are getting harder to find.
Verifiable tasks for long-context reasoning have to be long, diverse, hard, and checkable. Human annotation is expensive. Naturally occurring text that happens to contain a checkable multi-hop question is rare. Synthesizing it with a teacher works right up until the student catches the teacher, and then you have no teacher left.
Distillation has the mirror problem. You need a privileged expert that sees more than the student does. Privileged information gets scarce exactly when the task gets hard.
Six papers from this cycle attack different parts of that wall. The overlap between them is the story.
LongZero: write the answer first, build the document second
LongZero's move is a reordering. Standard synthesis writes a question over a document you already have. LongZero has the model write the question, the answer, and the supporting evidence in a compact structured form first, then expands that evidence into a long multi-document context.
The order matters. The fact you need is guaranteed to be present, because you wrote it before you wrote the context. You are constructing the dump around the fact rather than hoping a 60-page dump happens to contain it.
On top of that, the model writes multi-round retrieval tasks. Both task families train under GRPO.
Two models, five long-context benchmarks. LongZero beats RL on data synthesized from external corpora and generalizes to real tasks. No corpus, no teacher. That last part is the whole point: the only input is the model's own proposal distribution, which by construction never falls behind the model.
Context length becomes a generation knob. Want to train at 128K? Expand the evidence until you hit 128K. You don't need to find a 128K document that contains the right fact.
ASTEROID: distillation that teaches a student to explore
Partial observability is where imitation quietly breaks. An agent that can't see the hidden state has to gather information before it can act. A privileged expert with full state never needs to search, so its demonstrations contain no search behavior. Train a student offline on those demonstrations and you get the average expert action across every hidden state that could have produced a given observation. That average is usually a bad action.
Quick Take: All six papers get their gains by changing what the model is asked to compare or what the gradient is measured against, not by collecting more comparison data.
ASTEROID's fix is small, and it took me a while to see why it works. The student runs in the environment and collects its own on-policy history. Then the privileged expert labels actions conditioned on that history, not on the environment state. Because the student's history reveals how confused it is, the expert's labels end up encoding "you don't know yet, so go look."
Eleven environments, spanning discrete exploration benchmarks, robotic manipulation, navigation, and Procgen. Roughly 2x higher returns under a fixed exploration budget, roughly 100x fewer interactions than online RL, and it beats frontier models on the exploration environments it was tested against.
The 100x number is the one that matters operationally. Sample cost is the reason most teams never train exploration behavior in the first place. Policies trained in simulation also transfer to a real robot with vision denied, adding 40% success over standard distillation, which means the behavior survives the sim-to-real gap rather than being an artifact of a toy benchmark.
When your two objectives cancel: CORD
Joint RL plus self-distillation is the standard recipe for agents. Task reward for the outcome, dense teacher guidance for the process. Those two gradients can point in opposite directions, and a mixing coefficient only rescales them. If the RL gradient and the distillation gradient oppose each other, no scalar weight makes the conflict disappear. It changes who wins by how much.
CORD treats it as a projection. Pick one objective to protect. Combine the gradients once per optimizer mini-batch. If the combined step would move you against the protected objective, project onto the nearest point that doesn't. The nearest-point formulation has a closed form, so the cost is a single projection per mini-batch rather than a per-token correction.
The design detail I like: CORD separates which objective you protect from what triggers the projection. Those are two independent choices, and the paper argues you have to make them together. Protect distillation and gate on RL conflict, or the reverse, and you get different failure modes.
Results come from three existing joint-training recipes, unchanged except for the gradient combination. ALFWorld success improves 1.5 to 15.9 percentage points across them. WebShop improves in two of three pairs, with SDAR going from 60.7% to 64.6%.
Look at the spread rather than the headline. A 1.5 point floor means the projection helps most where the two gradients actually fight. If your recipe never had conflict, CORD is dead weight with an extra knob.
EPIC: token-level clipping measures the wrong thing
This is the paper that made me re-read my own training code.
PPO clipping constrains the ratio of new to old policy probability for the current token. But in autoregressive generation, the prefix that token was conditioned on was also sampled by the policy. If the prefix drifted, the token ratio doesn't see it. Over a long response, or under off-policy sampling, the drift compounds across hundreds of tokens while the per-token ratio sits comfortably inside the clip range.
EPIC reformulates generation as a finite-horizon Markov decision process and derives the performance-difference identity under behavior-policy sampling. The exact importance weight for each stepwise improvement term is the cumulative likelihood ratio of the generated prefix, not the single-token ratio. Change of measure does the work: the thing that actually changed is the distribution over prefixes.
From there the trust region is built on the exact reverse-KL coordinate r − 1 − log r, whose expectation under the behavior policy equals the prefix-distribution KL divergence. You get a trust region on prefixes with no local quadratic approximation and no routing replay.
The experimental setup is the hard case: off-policy RL with a sparse mixture-of-experts language model. EPIC beats standard token clipping and sequence-level clipping across mathematical reasoning benchmarks, and stays competitive with routing-replay alternatives that cost more to run.
If you train multiple epochs over the same rollouts, or with any data that isn't strictly on-policy, the clip range you set is not the constraint you think it is.
Key Numbers
- 100x fewer environment interactions for ASTEROID than online RL baselines
- +40% real-world manipulation success from simulation transfer, over standard distillation
- 15.9 percentage points is the best ALFWorld gain CORD reports across three recipes
- 60.7% → 64.6% WebShop success for the SDAR recipe with CORD
- s* ≪ d is the assumption that makes private RLHF error scale with the sparse feature count instead of the model dimension
Uni-Mem: memory construction and retrieval want the same policy
Agents with long-term memory usually run two stages. Build the memory store, then retrieve from it at query time. Teams optimize one or the other. Optimize construction and retrieval stays frozen. Do RL on retrieval with offline-supervised construction and the two stages never co-adapt.
Uni-Mem puts both under one policy and trains end-to-end online. The obstacle is structural. GRPO's group-relative advantage assumes rollouts in a group differ along the axis you care about. With a unified memory manager, rollouts differ in construction quality and retrieval quality at once, so the comparison is confounded and feedback over a long memory-management trajectory is sparse.
Stage-Decoupled GRPO alternates construction and retrieval optimization while holding the other stage's variation constant inside each rollout group, which gives each stage a clean learning signal. Event-Wise Credit Assignment then localizes feedback to the memory events responsible and propagates event-level advantages to the operations that produced them.
On LoCoMo and LongMemEval, Uni-Mem beats the strongest RL-based memory baseline. The lesson travels past memory: any time you put two learnable stages under one GRPO loop, group baselines stop being comparable unless you control the other stage.
Privacy: keep the noise proportional to the sparse part
Private RLHF with dense reward parameterization gives estimation error that scales polynomially with the ambient dimension d. When d is a few billion, that bound tells you nothing useful.
The sparse version assumes the reward depends on s* features, with s* much smaller than d. Under squared L2 parameter error, the upper bounds are Õ(s*/n + s*²/(n²ε²)) for (ε,δ)-sample-level DP and Õ(s*/n + s*/(nε²)) for ε-label-DP with randomized response, where n is training data size. The lower bounds show the sample-DP rate is nearly optimal and that the label-DP guarantee cannot be improved inside the randomized-response framework.
Read the two rates side by side. The label-DP bound pays a single power of s* on top and puts ε² in the denominator. The sample-DP bound pays s*². If you're picking a mechanism for preference data, label-DP wins in this regime, and it also matches how you'd actually deploy it: privatize the label, not the feature vector. Empirically the framework beats dense baselines on sentiment generation and safety alignment.
Side by side, the six papers look like this:
| Paper | Target | Core mechanism | Headline result |
|---|---|---|---|
| LongZero | Long-context reasoning | Self-written multi-hop + multi-round retrieval tasks, GRPO | Beats RL on externally synthesized data with zero external data |
| ASTEROID | Test-time exploration | On-policy student history, expert labels conditioned on it | ~2x returns, ~100x fewer interactions, +40% sim-to-real |
| CORD | Joint RL + self-distillation | Closed-form gradient projection with a protected objective | ALFWorld +1.5 to 15.9 pp across three recipes |
| EPIC | PPO trust region for LLMs | Prefix-level exact KL clipping via r − 1 − log r | Beats token and sequence clipping on math reasoning |
| Uni-Mem | Agent long-term memory | Stage-decoupled GRPO + event-wise credit assignment | Beats strongest RL memory baseline on LoCoMo and LongMemEval |
| Private Sparse RLHF | Private alignment | Sparse reward estimation under Bradley-Terry-Luce | DP error scales with s*, not model dimension d |
What I ran into reproducing these
The GRPO confound in Uni-Mem bit me first, in a different setting. I had a two-stage pipeline where one module scored candidate retrievals and another rewrote the query, trained jointly with group-relative advantages. Training loss fell and eval stayed flat for a week. The rollouts in each group were varying along both stages at once, so the advantage was averaging over two axes. Splitting the stages and holding one constant fixed it in an afternoon.
The EPIC observation is easier to verify than to fix. Log your per-token clip fraction next to the cumulative prefix ratio on any run with more than one epoch and you'll find plenty of steps where the token ratio sits at 0.95 and the prefix ratio is at 3. You can see it with a few hundred lines of instrumentation. Fixing it means changing the trust region, which is the paper.
LongZero's reordering is also less trivial than it sounds when you try it. Writing the evidence before the context means your generation pipeline has a dependency edge where the standard pipeline has none. You can't parallelize the two halves. That's the cost, and it's small next to buying annotated long-context data.
Common Pitfalls
- Cranking the mixing coefficient when the joint loss stalls. If two gradients cancel, a larger weight on distillation doesn't remove the cancellation. It changes which objective loses. Project the conflict or don't mix the objectives.
- Computing the importance ratio from the single-token probability under off-policy data. The per-token ratio can look well-behaved while the prefix has drifted by a factor of three. Instrument the cumulative prefix ratio before you trust a clip range.
- Imitating a privileged expert offline and expecting exploration to appear. It can't. The expert never searched, so its labels contain no search. You need on-policy student histories for the labels to carry uncertainty.
- Synthesizing long-context tasks by dumping documents into a prompt and asking for a question. You lose the guarantee that the evidence is actually in the context, and you inherit the corpus's topic distribution along with it. Generate the evidence first, expand second.
- Letting two learnable stages share one GRPO group without controlling variation. Group-relative advantages assume a single axis of variation. Give them two and the learning signal averages to something that trains neither stage.
One Thing to Remember
Every method here changes what the model is asked to compare, not how much data it compares over. LongZero changes what the question and the context are relative to each other. ASTEROID changes what the expert is conditioned on. CORD changes how two gradients combine. EPIC changes what the ratio measures. Uni-Mem changes what a rollout group holds constant. Sparse RLHF changes which parameter count enters the error bound. Structure first, volume second.
The Bottom Line
If you're training long-context reasoning and you don't have a teacher stronger than your student, adopt LongZero's ordering: write the verifiable core first, expand the context afterward. You keep the guarantee that the answer is supported, and context length turns into a generation parameter instead of a data-collection problem.
If you're joint-training RL with self-distillation and the loss plateaus, add CORD's gradient projection before you tune the mixing weight. It's closed-form and costs one operation per mini-batch. If a quick diagnostic shows your two gradients barely conflict, skip it and spend the effort elsewhere.
If you run PPO or GRPO for more than one epoch, log the prefix-level importance ratio this week. Token clipping under off-policy data is the quiet failure in this batch of papers, and EPIC is the fix once you've confirmed the drift.