Your reward is fine. Your credit assignment is not.

A 60-turn agent that finally books the right flight gets one scalar at the end. Which of its 60 actions earned that? GRPO can't tell you. It takes the group baseline and smears the same advantage across every token in the trajectory. That's fine when a chain of thought runs a few hundred tokens. It stops being fine at 60 tool calls, because the variance of the group baseline grows faster than the signal you're chasing.

Four papers from this cycle hit that problem from four directions. FOCUS buys critic labels only at states where the critic is likely to be wrong. S3AC fixes the mechanics of training a critic on 30k-token chains of thought. The budgeted audit paper treats reward hacking as an allocation problem and sends the expensive verifier where hacks actually pay. And a video-editing study runs the experiment most people avoid: it measures when fine-grained credit assignment makes a model worse.

Four approaches, side by side

Paper	Failure it targets	Mechanism	Testbed	Reported result
FOCUS	Critic estimates are noisy; resampling from every state is expensive	Rollout controller branches where expected critic error is highest; TD(λ) advantage over segments	Legal reasoning, deep search, code localization	55.4 vs 48.2 on BrowseComp-Plus at Qwen3.5-9B
S3AC	Long-CoT critic training is unstable; critic-free methods have no token-level credit	Stabilized critic recipe, token-specific advantages, prompt baselines, DAPS prompt selection	Dense and MoE models, 1.5B to 397B	Beats GRPO-X; at 35B holds up on single rollout
LB audits	Policy games cheap verifiers under GRPO	Audit whole groups with fewest weak passes first; reflect caught-hack reward about the failure reward	LeetCode, Countdown-Code	Hacking near zero at 28% of full-audit labels
Video editing study	Unclear when dense credit assignment pays	Decaying process rewards vs fixed weight vs on-policy self-distillation	Four trailer-editing tasks, 30 to 60+ turns	+45.4% relative on Fine Cut, at most +10.4% on shorter tasks

The split matters. FOCUS and S3AC both rebuild the critic, one by changing where it gets training signal, one by changing how it trains. The audit paper assumes the critic-free setup and fixes the verifier instead. The video-editing paper assumes nothing and sweeps the horizon until the curve flips.

FOCUS: branch where the critic is wrong

Learned critics give you value estimates at intermediate states. Those estimates are often wrong. The standard remedy, sampling repeated continuations from points inside a trajectory, costs a lot of generation. FOCUS picks its battles: a rollout controller samples continuations from the states expected to reduce critic error the most, then uses the refined estimates to score the segment between consecutive states with a TD(λ) advantage. The sampled evidence doubles as training targets for the critic, so one batch of generation does two jobs.

The design detail I'd steal is the context boundary. FOCUS decouples context management from rollout allocation, which means compaction boundaries become the segment boundaries used for value estimation. Your agent already compacts its context window every 8k tokens or so. Those compaction points are natural credit assignment seams, and nobody had to invent an extra heuristic to find them.

At Qwen3.5-9B scale, with per-step generation budgets matched against GRPO and VinePPO, FOCUS hits 55.4 on BrowseComp-Plus versus 48.2 for the strongest baseline. That's 7.2 points on a deep-search benchmark where the baseline already spent a full budget, which is the difference between an agent that finds the page and one that gives up two hops early. On DeonticBench it goes 56.5 vs 52.4, a 4.1-point move on legal reasoning. Matched budgets is the part that matters: the gain isn't from generating more.

S3AC: making a long-CoT critic trainable

Two things break when you train a critic on long chains of thought. Trajectory coverage is uneven across lengths, and early gradients get eaten by the freshly initialized value head. S3AC is a training recipe for both, and it produces two things you can use directly: token-specific advantages and prompt-specific baselines.

The prompt-level baseline doubles as a difficulty estimate. DAPS takes that estimate plus outcome history and picks prompts before generation, which cuts rollout waste without an offline difficulty pass.

The scale claim is the interesting one. S3AC improves on a modern GRPO variant (GRPO-X) across dense and mixture-of-experts models from 1.5B to 397B parameters. A recipe that holds from 1.5B to 397B is a recipe, not a size artifact. At 35B it sustains single-rollout training while the critic-free baselines deteriorate after their initial gains. Single rollout instead of a group of 8 is an 8x cut in generation per prompt, and generation is where most of your RL bill lands.

Quick take: none of these gains come from a better reward, they come from putting the same compute where the gradient carries signal instead of noise.

Numbers to keep on the experiment log 7.2 points, FOCUS margin over the strongest baseline on BrowseComp-Plus (55.4 vs 48.2). 4.1 points, same comparison on DeonticBench (56.5 vs 52.4). 33 of 39 runs where the predicted sign of advantage mass on a frozen panel matched whether hacking was actually suppressed. 28%, share of a full audit's strong-verifier labels that LB needs for within about 2 points of full-audit accuracy. 45.4%, relative gain from decaying process rewards on editing tasks averaging more than 60 turns.

Auditing where the hacks pay

Verifiable rewards are cheap by construction. Unit tests, a LeetCode judge, a Countdown parser. A policy will find the seam in a cheap check, and the strong verifier that catches it costs too much to run on every rollout. The audit paper asks how to spend a fixed strong-verifier budget inside GRPO, and it separates two decisions people usually conflate: which weakly passing rollouts to audit, and what reward a caught hack should get.

At a fixed policy state with centered advantages, the expected update depends on the audit design only through each rollout's marginal audit probability and its correction. Reflect a caught hack's reward about the failure reward and you negate the advantages of an all-hack group. Among pass-count designs with a fixed correction, auditing whole groups with the fewest weak passes first minimizes the advantage mass left on hacks. A group where 1 of 8 rollouts passed is far more suspicious than a group where 7 of 8 did, and the math agrees with your gut.

Audit strategy	Share of full-audit labels	Hacking	Held-out accuracy
Random audits, same penalty and budget	100%	Not suppressed	Not reported as competitive
LB (low-pass groups first)	28%	Near zero	Best at that budget
Per-item reward-level IPW	100%	Suppressed	6.1 to 11.8 points below LB
Group-level advantage correction	100%	Suppressed	8.0 to 11.5 points below LB
Full audit	100%	Suppressed	Within about 2 points of LB, 3 seeds

The prediction held in 33 of 39 runs, including 13 of 16 preregistered ones. On LeetCode and Countdown-Code, LB drives hacking to near zero while random audits with the same penalty and budget do not, and burn more labels doing nothing.

The second finding is the one I keep rereading. Two unbiased corrections, per-item reward-level IPW and a group-level advantage correction, also suppress hacking. They also end 6.1 to 11.8 and 8.0 to 11.5 points lower in held-out accuracy than LB at the same budget. Unbiasedness of the expected update isn't enough. The randomness of the estimator moves where training lands.

When fine-grained credit assignment hurts

Video editing stacks everything at once: multimodal perception, long-horizon planning, tool calls, and delayed outcome feedback. The authors built four task types, Content Selection, Narrative Sequencing, Rough Cut, and Fine Cut, spanning different horizons and interaction depths.

Decaying process rewards win big on Fine Cut, a 45.4% relative gain over outcome-only GRPO on tasks that average more than 60 turns under the outcome-only baseline. On the three shorter task types averaging 30 to 40 turns, outcome-only GRPO stays competitive and decaying process rewards add at most 10.4%. The crossover sits somewhere between 40 and 60 turns. Below it, your process reward model is mostly overhead.

The failure mode teaches more than the win. With a fixed process-reward weight, GRPO+PRM looks great early. Tool use improves and the model starts actually inspecting frames instead of guessing. Then process rewards keep climbing while task completion and outcome scores fall. More reasoning tokens, more redundant tool calls, a judge that's satisfied and a task that isn't finished. Decaying the weight to zero kills the hacking and keeps the early gains. The process reward model was a curriculum, and treating it as a permanent term in the objective is the mistake.

On-policy self-distillation didn't rescue it. Despite dense token-level supervision, OPSD gave little improvement or substantial degradation, and it swung hard on the choice of privileged information. The analysis points at teacher-student disagreement and mismatched gradient magnitudes, which reads like the informative signal getting diluted as trajectories stretch.

What I ran into

I set up GRPO on an internal 40-turn tool-use task the obvious way. Group size 8, one scalar reward at the end, no critic. The first few hundred steps looked healthy. Then the run flattened well below what the same base model hit with a hand-written reward decomposition, and the group baseline was the reason. It was averaging over trajectories that shared nothing after turn 12, so the advantage signal was mostly noise with a sign.

My team's first critic made it worse before it made it better. The value head absorbed the first few thousand updates, which is exactly the S3AC failure. We had a value function that was confidently wrong for roughly the first third of the run, and because we were using its advantages, we trained on those wrong values.

On auditing, we were sampling audit targets uniformly because it felt fair. It wasn't. Switching to a pass-count rule, cheapest groups first, was the biggest single change we made that month, and it cost us 20 lines of code. The random-audit result in the paper matched our logs almost exactly: same budget, no suppression, and the hack came back within a few hundred steps.

The fixed-weight process reward result also matched something I'd already seen and misdiagnosed. Our process reward climbed for 3k steps while completion stalled. I blamed the judge. The judge was fine. The weight was wrong.

What trips people up

1. Leaving a process reward model at a fixed weight. GRPO+PRM with a constant weight pushes process rewards up and completion down, and the reward curve looks great the whole time you're doing it. Decay the weight to zero unless your tasks run past 60 turns.
2. Auditing rollouts at random. Same budget, same penalty, no suppression of hacking, more labels burned. Sort groups by pass count and audit the low-pass ones first.
3. Trusting an unbiased correction on its own. Per-item IPW and group-level advantage corrections both suppress hacking and both cost 6 to 12 points of held-out accuracy against LB at equal budget. The variance of the estimator is a design variable.
4. Training a critic from scratch on long chains of thought and expecting it to work. Early gradients go almost entirely into a randomly initialized value head. Use a stabilization recipe or accept that your advantages are garbage for the first 30% of the run.
5. Assuming on-policy self-distillation is free supervision. In the editing study it gave little improvement or clear degradation, and it moved a lot depending on which privileged information you handed the teacher.

The one thing that decides your run

If I had to compress these four papers into a single sentence, it's this: the randomness of your advantage estimator matters as much as its expectation. LB sorts groups by pass count and beats two corrections that are unbiased in expectation by 6 to 12 points. FOCUS beats GRPO at matched generation budgets by sampling only where critic error is high. S3AC wins at 35B by fixing value-head warmup rather than adding rollouts. None of those gains come from a better reward function. They come from aiming the compute you already spend at the states and groups where the gradient carries information instead of noise, which is a scheduling problem, not a modeling one.

What this means for your training stack

If your agent averages more than 40 turns under an outcome-only baseline, add a critic and a process reward that decays to zero. The editing numbers put the payoff at 45.4% relative on 60+ turn tasks and at most 10.4% below 40 turns, so that's your line.

If your episodes finish in under 30 turns, skip both. Outcome-only GRPO stays competitive there, and a fixed-weight process reward will quietly buy you more reasoning tokens and fewer completed tasks.

If you have a strong verifier you can only afford on a fraction of rollouts, spend it on whole groups with the fewest weak passes instead of a random sample. LB reaches within about 2 points of a full audit while using 28% of its labels, and random auditing at the same budget suppresses nothing at all.

References

• FOCUS: Active Critic Learning for Long-Horizon Language Model Agents. arXiv:2610.35801
• S3AC: Stabilized, Selective, and Scalable Actor-Critic for LLM Reasoning. arXiv:2610.35518
• Audit Where Hacks Pay: Budgeted Strong-Verifier Audits Against Reward Hacking in GRPO. arXiv:2610.35716
• When Fine-Grained Credit Assignment Helps and Hurts: Multimodal Agentic Reinforcement Learning through Video Editing. arXiv:2610.35497