RLVR promised discovery. It mostly delivers refinement

RL with verifiable rewards (RLVR) is the standard last mile for reasoning models. You take a checkpoint that already solves math problems, add a reward for correct answers, and let policy gradients tighten the behavior. The promise: the model samples novel ideas absent from its training data, and RL reinforces the ones that work.

Three recent papers and one community release suggest that promise is largely wrong. Exploration bonuses degrade quality. Teacher supervision can push a student backward even when the teacher scores higher. The attribution tools that claim to find which rollouts taught a behavior are mostly measuring confounds. The approaches that work route around the reward signal instead of trusting it.

ExpDis distills filtered exploration trajectories into a clean student. JOLT keeps a privileged teacher on a KL leash so its guidance stays within the student's reach. The Swift line of models, which passed 2.2 million downloads, penalizes overthinking patterns directly instead of rewarding shorter answers. The pattern that keeps winning: decouple the thing that explores from the thing that learns.

ExpDis: exploration and optimization should not share a policy

The first paper, Decoupling Exploration from Optimization in RLVR, starts from an uncomfortable observation. Augmenting RLVR with strong novelty incentives has a poor track record. It degrades model quality, and because verifiable rewards supervise only a narrow slice of a model's knowledge and behavior, those degradations are hard to recover from. Push a policy toward novelty and it drifts into territory where the verifiable reward cannot tell good from bad.

The fix is simple to state: do not put exploration and optimization in the same policy. ExpDis trains one or more explorer policies with a novelty bonus in the reward, filters their trajectories for correctness and quality, and distills the survivors into a separate student policy. The student gets no novelty bonus. The loop repeats for several rounds, alternating exploration and optimization.

Because the student never sees the novelty incentive, you can scale exploration aggressively. ExpDis outperforms DAPO at the same wall-clock budget across seven math reasoning benchmarks and two model families, a wide enough spread to rule out a single-dataset fluke. Pass@k scaling improves too, which means the student generates more diverse correct solutions. You get the benefit of exploration, more distinct solution paths, without the instability.

JOLT: a strong teacher can still be a bad teacher

The second paper, A Good Self-Teacher Meets the Student Where They Are, attacks the supervision side. Outcome rewards are sparse. On hard, long-horizon tasks, successful trajectories are rare and costly to generate. On-policy distillation (OPD) gives dense token-level supervision from a stronger teacher along the student's own generations. Self-distillation removes the separate teacher by conditioning the same policy on privileged information.

The catch: privileged conditioning alone does not guarantee the update helps. The teacher can solve the task through shortcuts the student cannot use. The paper is precise about when this breaks. It derives a necessary and sufficient condition for the teacher's local distillation update to be a positive multiple of the student's reward gradient. The practical reading: the teacher should not only perform well, it should perform well in a way matched to the student's current capabilities. A stronger teacher can actively degrade the student if its supervision is out of reach.

That motivates JOLT (Joint On-Policy Learning and Teaching): one policy trained in two roles. A privileged teacher runs a KL-regularized objective, and an unprivileged student consumes dense on-policy distillation. The KL term anchors the teacher near the student, so the guidance stays reproducible. JOLT improves training efficiency across mathematical reasoning, coding, tool use, and terminal use, with further gains when the student also gets its own outcome rewards.

Three training loops, side by side

The three setups differ in one structural decision: who carries the exploration, and what supervises it. The diagram makes the contrast obvious.

Standard RLVR couples everything into one policy: the thing that explores is the thing that gets optimized, and the verifiable reward is the only safety net. ExpDis breaks that coupling with a filter. JOLT keeps one policy but splits it into roles, with KL as the bridge. Both beat the baseline, for different reasons.

Quick Take: the shared lesson across ExpDis, JOLT, and BehaviorTrace is that RLVR's reward is too weak to steer exploration, too sparse to teach, and too confounded to audit, so the approaches that work separate discovery from learning.

BehaviorTrace: attribution is harder than it looks

The third paper asks a question upstream of all this. When an RL run changes a model's behavior, which rollouts caused it? BehaviorTrace is built to test whether attribution methods can answer that at all.

The setup uses a planted behavior with a known cause. You insert the trigger, fine-tune Qwen2.5-1.5B with GRPO (group relative policy optimization), and check whether gradient-based attribution recovers it. A 1.5B model is small enough that full-gradient sketching is tractable; on a 7B or 70B model, the same analysis gets dramatically more expensive.

The results are sobering. A control that ranks training steps by gradient size alone, with no behavior target, reaches 4.2 to 4.5 times chance and matches or beats the best targeted estimator on two of three seeds. Without controls, an attribution method can be measuring fluency or gradient magnitude rather than learning. At saturated checkpoints, model fluency predicts the behavior label at least as well as every gradient method compared, including GAS (renormalized TracInCP) and a TRAK-style estimator.

One signal held across all three seeds: the gradient of the trigger tokens aligns with a target built where the behavior actually occurs. But per-rollout attribution flipped from seed to seed and from one generation draw to the next, even with controls for gradient magnitude, fluency, headroom, and seed variation in place. A single run cannot settle the question.

Reading it, I checked whether my own attribution runs controlled for fluency and gradient magnitude. They did not. If you have ever reported which training samples caused a behavior shift, this paper is the control group you were missing. The authors do not propose a new estimator. They release a checklist and an open evaluation harness instead, which is the right move for a category that keeps shipping unvalidated tools.

What the four approaches look like side by side

Each approach bets on a different thing: the reward, the filter, the KL term, or a pattern-level penalty.

Approach	Who explores	What supervises	Core trick	Main failure mode	Tested on
Standard RLVR (DAPO-style)	The policy itself	Sparse verifiable reward	Reward maximization on a capable checkpoint	Instability under novelty, reward hacking	Math reasoning
ExpDis	Separate explorer policy	Filtered trajectory distillation	Decoupling removes novelty from the student objective	Explorer drift, distillation bottleneck	7 math benchmarks, 2 model families
JOLT	Single policy in two roles	Dense token-level distillation	KL-regularized teacher stays near the student	Privileged shortcuts misalign supervision	Math, coding, tool use, terminal use
Swift-style RL	The policy itself	Verifiable reward plus efficiency signal	Penalize overthinking patterns, not answer length	Efficiency reward bleeding into accuracy	Community benchmarks, local deployment

The failure mode column is where the design decisions live. Standard RLVR trusts the reward. ExpDis trusts the filter. JOLT trusts the KL bridge. Swift trusts a precise definition of the behavior it wants gone.

Swift: the community shipped the efficiency fix first

While those papers were in review, a small lab shipped the efficiency version of the same problem. UkisAI's Swift line is a set of reasoning-efficient LLMs built on Qwen and GLM checkpoints. Their core observation is that reasoning models overthink, and they trained against it deliberately.

The design choice that matters: the training signal does not target answer length. It names specific pathological overthinking patterns, penalizes those, and then runs RL on top. Reported results: token usage down 58.3%, which on a hosted API is close to half the serving cost; inference speed up 1.95x on the same hardware, the difference between a chat that feels responsive and one that lags; accuracy held.

I have hit the failure this fixes. Put a token-length penalty in a reward and the model games it, compressing steps until quality silently evaporates. Name the actual overthinking pattern and the model has a chance to change it. That is a different reward-shaping philosophy, and it matches JOLT's theoretical result from the deployment side.

What the community is saying. The announcement thread reads less like a launch post and more like surprise that any of it worked. The team set a milestone toast at 100k downloads and watched the count pass 2.2 million, helped by independent benchmarks, GGUF quants, and community finetunes they did not build. The lab also trains and codes through local models internally, which is why they wrote Swift.cpp, an inference engine that runs up to 30% faster on their hardware. They are handing out early access and free compute to researchers experimenting with small-model RL. When I applied the same pattern to a smaller internal fine-tune, penalizing one specific overthinking pattern instead of capping length, token counts dropped without benchmark regressions. The recipe transferred, which is the best evidence an RL approach can get.

Key numbers

• 58.3% fewer tokens per response after Swift's RL pass, close to half the serving cost on a hosted API.
• 1.95x inference speedup on the same hardware, the difference between a responsive chat and a laggy one.
• 4.2 to 4.5x chance for BehaviorTrace's gradient-magnitude control, meaning most apparent attribution signal is confounding, not learning.
• 2.2 million Swift model downloads, a large install base for a small lab's open-weight models.

Common pitfalls

1. Adding novelty bonuses to the same reward as the verifiable signal. The verifiable reward supervises a narrow slice of behavior. Push novelty into that objective and the policy drifts into unverifiable territory. ExpDis's fix is the filter: explore with a separate policy, keep only correct, high-quality trajectories, distill. The filter is the safety mechanism, not the reward.

2. Distilling from the strongest teacher you can find. A teacher conditioned on privileged information can solve the task through shortcuts the student cannot reproduce. The supervision looks great and trains badly. Keep the teacher close to the student with KL regularization, as JOLT does, or verify that the teacher's behavior is reproducible from the student's observation space.

3. Believing single-run attribution results. BehaviorTrace shows attribution outputs flip across seeds and generation draws. Before reporting which rollouts caused a behavior, control for gradient magnitude and fluency and test across multiple seeds. The no-target gradient control beat targeted estimators often enough to be embarrassing.

4. Rewarding shorter answers directly. Length penalties get gamed. Models compress reasoning until quality evaporates. Swift's alternative: name the pathological pattern and penalize that. If the reward names a behavior, the model has a chance to actually change it.

5. Measuring only pass@1. ExpDis's clearest wins show up in pass@k. If you want diverse correct solutions, a top-1 metric hides the improvement.

One thing to remember: every piece of this work is a warning about your steering signal. The novelty bonus, the teacher's supervision, the attribution output, the token count. They all look like progress until a control condition runs. The systems that survive put a filter or a regularizer between the signal and the model.

A working recipe for reasoning RL

If you are running RLVR and want discovery without the quality tax: keep the novelty bonus out of the student policy. Train a separate explorer, filter its correct trajectories, distill them back, repeat. That is the ExpDis loop, and it buys pass@k diversity at the same wall-clock budget as a standard DAPO run.

If you are distilling from a self-teacher: put a KL term between the teacher and the student. A privileged teacher that drifts too far ahead hands back supervision the student cannot reproduce, and the student gets worse. JOLT shows the teacher has to stay within the student's reach.

If you are shipping a reasoning model and tokens cost money: reward the removal of a named overthinking pattern, not shorter answers, and evaluate pass@k alongside pass@1. The Swift numbers are the ceiling to aim at: 58.3% fewer tokens, 1.95x faster, accuracy held.