Agentic RL Is Wrestling With One Problem: Who Gets Credit?

The bottleneck moved from models to credit assignment

RLVR did something clean for single-turn math. Generate a batch, check the answer, lift every token in a correct response, push down every token in a wrong one. That works because the trajectory is short and the reward is unambiguous.

Multi-turn agents break the assumption. A ten-step web navigation episode collapses into one final success flag. GRPO broadcasts that flag to all ten steps equally. So a structurally invalid action sitting inside a winning trajectory gets reinforced as hard as the step that actually moved the task forward. That is the wall this cluster of eight papers keeps hitting from different directions.

Most of these papers are fixing GRPO, not replacing it. The interesting work lives in where the credit lands, and how confident you are when you hand it out.

Paper	Mechanism	Testbed	Headline number
Two Clocks	Batch and learning-rate scaling theory	Qwen2.5-Math-1.5B, 7B	Progress gap persists across a 16x batch range
ProAS	Reference-graph credit shaping	Qwen2.5-1.5B/7B, ALFWorld, WebShop	94.3% ALFWorld success
Proximal KL	Token-wise off-policy correction	Qwen3-1.7B/4B, eight math benchmarks	+2.58% over GRPO
SkillScaffold	Selective skill injection	ALFWorld, WebShop, ScienceWorld, tau2-Bench	Gains on long-horizon tasks
Confidence Calibration	Outcome-grounded step confidence	Qwen2.5-7B, ALFWorld, TextWorld, WebShop	AUROC 0.68 to 0.85
Self-distillation	Reality check on SDPO and OPSD	Qwen3 1.7B to 14B, OLMo-3-7B	Gains don't generalize
DDRG	Graph-aligned verification	Qwen2.5-7B, Llama3.1-8B	+8.65 and +15.83 points
SRO	Task-adapted recipe search	Qwen3-1.7B/4B	53.8 vs 40.8

Two clocks: bigger batches, slower learning

The first paper asks the question every team running RLVR eventually asks. If I double the rollout batch, can I double the learning rate and keep progress? The square-root rule says roughly yes, and it's been reported to preserve progress per sample with one optimizer step per rollout.

The authors show the rule breaks under multi-step training. They split training into two clocks. One counts samples needed to make progress. The other counts samples consumed before the run collapses. A local quadratic model shows the square-root rule preserves first-order drift per sample but not actual progress. Bigger batches take fewer, longer updates, and each long update loses more to curvature. Past a learning-rate-dependent critical batch, sample efficiency drops.

That gives you a schedule you can use. Start below the critical batch, then switch to a larger batch before the failure clock hits. Ratio clipping delays the collapse, especially at large batches, but the progress gap returns once you remove it. Widening the clip buys a transient speed bump and costs stability. The gap shows up again on a different dataset, on a general-purpose checkpoint, and on a 7B model, so this isn't a math-benchmark quirk.

ProAS: stop rewarding the steps that didn't work

GRPO's uniform credit is the target here. ProAS keeps the critic-free setup and adds an Agentic Reference Graph that evolves during training. Two moves sit on top of it. Masking Unreasonable Incentives strips reward from structurally invalid actions inside preferred trajectories. Asymmetric Penalty Redistribution pays more attention to the early deviations in trajectories that eventually failed, because those early wrong turns cascade into everything after them.

On Qwen2.5-1.5B-Instruct and 7B-Instruct, ProAS pushes success to 91.9% and 94.3% on ALFWorld, and 66.7% and 70.8% on WebShop. Those ALFWorld numbers are near-ceiling for this class of task. It also converges up to 2.44x faster, which matters more than the peak numbers when you're the one paying for rollouts. The framework tolerates imperfect or distilled reference graphs, so you don't need a flawless structure to collect the benefit.

Quick take: none of these papers replaces GRPO, they just fix where its uniform advantage signal lands.

SkillScaffold: inject a skill only when it pays

Textual skills are supposed to help exploration, and the usual approach injects them whether or not they help the current policy on the current query. SkillScaffold makes injection conditional. It reuses outcome-verified training trajectories to generate candidate skills, then estimates each one's utility by comparing likelihoods with outcome awareness. No per-candidate environment rollouts.

The problem it solves is real. When I ran a skill-injection setup on a small eval set, the rollout-based utility estimates swung hard between runs, and adding more candidates made the signal noisier instead of sharper. SkillScaffold avoids that by scoring against outcomes it already has. If no candidate looks likely to help, it runs skill-free and abstains. During training a Generator produces skills tailored to the current Executor, and the training pressure pushes those skills inward so inference can drop them entirely. The gains hold on ALFWorld, WebShop, ScienceWorld, and tau2-Bench, on both short and long-horizon tasks.

Off-policy RLVR: correct tokens, not whole responses

Off-policy data shows up everywhere in production runs: async generation, rollout-training mismatch, response reuse. The standard fix, multiplying likelihood ratios across a whole response, blows up in variance. Token-wise importance weighting avoids that but biases the gradient.

The Proximal KL paper proves that full future KL credit makes the exact proximal optimizer the unique stationary policy of the update, even for mixtures of older policies, with no full-response weights and no fitted critic. That is a strong result on paper. The practical version separates the proximal reference from the sampling policy and replaces PPO clipping with a bounded future KL credit.

Across seven matched off-policy comparisons on Qwen3-1.7B and Qwen3-4B, spanning two learning rates, response reuse, and rollout staleness, it beats GRPO by up to 2.58% micro-averaged accuracy over eight math benchmarks. In periodic-refresh runs it avoids the collapses that hit plain policy gradient and CISPO, and reaches higher accuracy in fewer updates. Token-wise correction also beats sequence-level and unweighted controls in ablation. Truncating the KL credit to a finite future horizon improves AIME24 mean@16 and pass@16 over the full tail, because it reduces correction variance.

Key numbers from this cluster

• 2.44x faster convergence from ProAS, at the top end
• 94.3% ALFWorld success at 7B from ProAS
• +2.58% over GRPO across eight math benchmarks from Proximal KL
• step-level AUROC of 0.68 to 0.85 for failure prediction
• expected calibration error from 0.42 down to under 0.12 on live ALFWorld
• 41.1 vs 15.4 average score on Qwen3-1.7B from SRO

Common pitfalls

Do not scale your rollout batch past the critical batch and expect the square-root learning-rate rule to save you. The rule preserves first-order drift, not progress. Measure convergence, not raw steps.

Do not keep uniform advantage on multi-turn trajectories. If your reward is episode-level and your trajectory contains dead steps, you're training the dead steps in. Mask them or redistribute the penalty before you blame the base model.

Do not score self-distillation on the same distribution you trained on. The reported gains largely come from overfitting or from transferring reasoning the model already had. Held-out tasks are the only honest test.

Do not calibrate confidence against the correctness of the current step. Calibrate against the episode outcome, because that's the thing you actually care about when deciding whether to abstain or hand off.

Do not widen ratio clipping for a quick bump. It trades stability for a transient gain, and the progress gap comes back.

The self-distillation reality check

This one will annoy people. Self-distillation trains a model to imitate its own predictions when it has privileged information, like correct answers or environment feedback. SDPO and OPSD report better sample efficiency and generalization than RL.

The authors rerun both across code, science, tool use, and math, using Qwen3 models from 1.7B to 14B plus OLMo-3-7B, and find the gains don't hold on unseen problems. What looks like learning is mostly overfitting, or transferring reasoning the model already had. Worse, the self-distillation gradients show weak alignment with the policy gradient that optimizes expected reward directly. When your update direction barely correlates with the reward direction, the sample-efficiency story gets shaky fast. Using privileged information to improve reasoning is still an open problem.

Read this as a warning before you swap RL for self-distillation. It isn't a free lunch, and in-distribution eval will lie to you.

Letting the agent know it will fail

An acting agent has to decide, every step of a long-horizon task, whether to trust its next action, abstain, or hand off to a human. That decision needs a per-step confidence signal predicting whether the current rollout will eventually succeed. Verbalized confidence is the natural candidate, but existing methods calibrate it against the correctness of the current step, which is the wrong target.

The fix calibrates against the eventual episode outcome. Under GRPO the rollout group already supplies success labels, so a proper scoring rule grades each step's confidence against the realized result. In offline RL on Qwen2.5-7B across ALFWorld, TextWorld, and WebShop, step-level AUROC against episode outcome rises from near chance to 0.68 up to 0.85. An AUROC of 0.85 means that when the agent reports confidence, it's right most of the time, which is enough signal to gate execution. The top 10% most confident steps succeed about 4.3x more often than average.

Online training on the live ALFWorld agent pushes expected calibration error from 0.42 to under 0.12, and selective execution beats both the untrained agent and a format-only RL control without hurting task success. Where the gain lives matters. The agent learns from its very first step which tasks it will fail, while separating a succeeding attempt from a failing one on the same task is much harder.

Verification without a bigger trace bank (DDRG)

Self-consistency votes over final answers and leaves intermediate disagreements unresolved. DDRG takes a frozen bank of reasoning traces, builds source-grounded claim graphs, and aligns them to find claim-level conflicts. Those conflicts go to an answer-blind verifier. If a verifier-supported refutation reaches the majority answer's supporting path, DDRG triggers exactly one focused repair. Otherwise it returns the self-consistency vote unchanged.

Zero-shot DDRG improves macro-average accuracy over direct answering by 8.65 points for Qwen2.5-7B and 15.83 points for Llama3.1-8B. It also beats CoT, GoT, and GoV question-answering adaptations, plus self-consistency on that metric, without enlarging the candidate bank. The selective part is the point. It only repairs when it has evidence, so it doesn't introduce a new failure mode on traces that were already fine.

Automating the recipe (SRO)

Post-training is a pile of coupled decisions: which data sources, how to process them, how to compose domains, how to set the training config. All of it under a fixed experimental budget. Existing automation either searches a fixed recipe space or runs an open-ended agent whose edits stay implicit and can't be reused.

SRO changes the recipe space itself. A Recipe-Space Designer builds a task-adapted space, and a Multi-Domain Recipe Optimizer uses structured trial evidence to guide candidate construction and multi-objective Bayesian selection. In joint post-training across three capability domains and six benchmarks, SRO hits 41.1 vs 15.4 for the strongest baseline on Qwen3-1.7B with GPT-5.4, and 53.8 vs 40.8 on Qwen3-4B with DeepSeek-V4-Flash.

The 1.7B gap is the headline. A 2.7x jump at small scale says the baseline was leaving a lot of recipe quality on the table, and that at small scale the pipeline around the algorithm mattered more than the algorithm itself.

What This Means

Here's the one assumption the whole cluster leans on: you can generate correct rollouts and label them. DDRG only repairs when it finds verifiable evidence. ProAS needs a reference graph that can itself be imperfect. Confidence calibration learns from outcomes, so it needs outcomes you can afford to collect. When that assumption fails, everything downstream fails with it.

If you're running multi-turn agentic RL with episode-level rewards, add step-level credit shaping before you touch model size. ProAS-style masking of invalid actions and redistribution of early penalties gives you a 2.44x convergence win and near-ceiling ALFWorld success at 7B, cheaper than retraining a bigger base model.

If your rollouts are stale, async, or reused, don't patch it with PPO clipping. Switch to token-wise correction with bounded future KL, which beat GRPO by up to 2.58% across eight math benchmarks and dodged the collapses that hit plain policy gradient and CISPO.

If you're betting on self-distillation to replace RL for reasoning, hold off. The reported gains don't survive held-out evaluation, and the gradients weakly align with the reward direction. Revisit when someone shows privileged-information distillation beating RL on unseen tasks.

The quiet winner here is automated recipe search. A 41.1 vs 15.4 gap at 1.7B says the biggest near-term wins are in the post-training pipeline, not in a new loss function.