Agentic RL's Real Bottleneck Is Credit Assignment, Not Model Size

An agent that spends 40 tool calls on a search task gets one reward at the end. That scalar is supposed to tell the policy which of those 40 calls helped. It can't, and every paper in this cluster is a response to that failure.

I read these five back to back over a couple of evenings, and the pattern was hard to miss. MLExpert, KnowGPO, Mosaic, the steering-agent work, and MMSearch-OPD all replace the terminal scalar with something local, grounded in the environment, and comparable across trajectories. They disagree about what the local unit should be. An experiment window. A knowledge state. A session. A corrective intervention. A distilled token.

Key numbers: MLExpert lifts MLE-Bench Lite's average medal rate from 37.9% to 72.7% on a 35B MoE. KnowGPO's 8B search agent beats outcome-only RL by 3.8 to 4.2 points across Xbench, BrowseComp-ZH, and GAIA. Mosaic trains on Qwen3.5-4B and uses fewer sessions than trajectory-level advantage broadcasting. The steering agent transfers to executors it was never trained against.

The numbers are the least interesting part. The mechanism is where the transferable lessons live.

What these five systems actually change

System	Base model	Unit of credit	Tested on	Headline result
MLExpert	35B MoE	Multi-turn window between experiment results	MLE-Bench Lite, NatureBench Lite	37.9% to 72.7% average medal rate
KnowGPO	8B	Incremental knowledge gain per step	Xbench, BrowseComp-ZH, GAIA	+3.8 to +4.2 points over GRPO, GiGPO, IGPO
Mosaic	Qwen3.5-4B	Session, weighted by marginal contribution	BrowseComp	Beats advantage broadcasting with fewer sessions
Steering agent	jointly RL-trained	Intervention, split into fast and slow paths	Information-seeking search	Better performance and efficiency, out-of-domain transfer
MMSearch-OPD	not reported	Token-level teacher guidance, decayed over training	Long-horizon multimodal search	Gains from refinement, then from joint training after SFT

Look at the third column. GRPO shows up in four of the five systems, and none of them claim a better optimizer. The gains come from changing what gets compared against what, and which slice of the trajectory the reward sticks to.

Quick take: the optimizer was never the bottleneck; the unit of credit was.

MLExpert: turning a long MLE run into trainable windows

Machine learning engineering is the worst case for credit assignment. A single Kaggle run can involve dozens of experiments, and every result reflects all the decisions that came before it. The feedback is delayed and entangled. If a run wins a medal, was it the feature engineering at step 3 or the learning rate at step 27?

MLExpert stops treating the run as one trajectory. Between any two experiment results sits a window of multi-turn decisions: read the logs, switch the model, change a hyperparameter, submit. Windowed Interaction RL assigns each window a shared outcome-based reward. The training loop collects asynchronously and applies policy correction mid-trajectory, so the policy doesn't wait for a 12-hour run to finish before it learns anything.

Before any of that, hierarchical agentic SFT transfers general tool-use and coding ability into the MLE setting through outcome-guided supervision. The Kaggle-MLE Factory generates executable tasks with verifiable outcomes, which is the fuel that makes windowed rewards possible at all. No executable verifier, no windows.

Doubling the medal rate puts a 35B model in range of frontier systems like Kimi-K3 and GLM-5.3, which are considerably larger. The capability also showed up on NatureBench Lite, on agent scaffold optimization, and on plain algorithm optimization. A model trained to optimize ML pipelines picked up something about optimization in general.

KnowGPO: group trajectories by what the agent knows

Anchor process rewards to the surface state of a trajectory and deep search breaks you. At step 12 the agent has six messy web pages in context, two of them near-duplicates and one truncated. Two trajectories that look nothing alike in token space may have acquired exactly the same facts, and two that look similar may differ on the one fact that matters.

KnowGPO decomposes the information needed to answer a query into atomic knowledge conditions. The state of a trajectory at any step is the subset of conditions satisfied so far. Trajectories sharing a knowledge state get grouped for within-group comparison, and the incremental knowledge gain between consecutive steps becomes the process reward. The reward is grounded in the environment rather than in the model's own confidence signals, which is what makes it hold up when the retrieved context is long and noisy.

The 3.8 to 4.2 point improvement over GRPO, GiGPO, and IGPO is respectable. The second number is the one that matters commercially. Their 8B model rivals or beats most 32B open-source systems. A quarter of the parameters at equivalent accuracy means a quarter of the GPU memory and a materially cheaper serving bill, and for a search agent burning tokens on 30-turn investigations, that compounds fast.

Mosaic: match training to the inference you already ship

Context compaction and subagent forking mean your production agent already runs multi-session trajectories. Most RL recipes assume one continuously accumulated context. That mismatch is invisible until you train, evaluate, and wonder why the gains don't show up in production.

Broadcasting one trajectory-level advantage to every session is the default fix, and it's weak. It can't separate the session that found the key fact from the session that re-read the same page, and it gives the policy no signal about how many sessions a task should take.

Mosaic adds two levels to that picture. Trajectory-level advantage estimation incorporates session-usage signals. Session-level advantage is shaped by the estimated marginal contribution of each session. On BrowseComp with Qwen3.5-4B, it beats broadcasting in both the context-compaction and subagent-forking setups while using fewer sessions on average.

Fewer sessions is the part I'd watch. Every avoided session is a saved context window and a saved round of tool calls, and the paper frames session allocation itself as the next thing to optimize.

Steering agents: a cheap detector and an expensive fixer

Goal drift is what happens when an agent keeps executing competently toward a target it quietly stopped pursuing. Multi-agent systems make it worse, because drift in one agent propagates through handoffs and nobody notices until the final artifact is wrong.

The steering agent supervises execution and splits the job in two. The fast path decides whether to intervene and costs almost nothing per step. The slow path generates the correction and costs real tokens. Training both paths jointly with RL, using a pairwise ranking objective for the drift assessment plus GRPO for the intervention policy, is what turns the split into something useful rather than a latency decoration.

On information-seeking search tasks it improves both performance and efficiency for single-agent and multi-agent setups. Three properties matter for production. The learned steering generalizes out of domain. It transfers to executors it wasn't trained against, which means you can swap the underlying model without retraining the supervisor. And intervention frequency is controllable at inference without retraining. You can turn the steering budget up for a high-stakes research task and down for a latency-sensitive one, from the same checkpoint.

MMSearch-OPD: manufacturing hard multimodal questions

Multimodal deep search has a data problem at both ends. Hard questions are expensive to write, and process feedback on long multimodal investigations barely exists.

The Refine pipeline takes simple image-question pairs you already own and expands the reasoning chains around them. Parameterized transformations control how deep the search goes and how much cross-modal evidence the answer requires. You manufacture difficulty from data on hand instead of paying annotators for questions nobody can answer.

Training blends answer rewards and lightweight behavior penalties for RL with on-policy distillation. The reward gap modulates token-level teacher guidance, so the teacher speaks up mainly when the student goes wrong and the outcome is poor. An exponentially decaying distillation coefficient reduces teacher dependence as training proceeds. Reported gains come from refinement first, then from joint training after SFT. The decaying coefficient is the transferable detail. Distillation on imperfect student trajectories helps early and caps the ceiling if you leave it at full strength.

What trips people up

Serving multi-session execution while training on single-session trajectories is the most common version of this mistake, and it's silent. If your agent compacts context or forks subagents in production, your training data has to represent that. Otherwise you're optimizing a policy for a rollout structure that never happens at inference.

Grouping process rewards by surface state is the second. In deep search, long noisy contexts mean two trajectories with identical knowledge look completely different token-wise. Group by what was learned, not by what was retrieved.

Third, keeping a fixed distillation coefficient. Teacher guidance on a flawed trajectory helps early and limits you later. Decay it, and measure when the teacher stops earning its tokens.

Fourth, training a critique or steering model to intervene whenever it detects drift, then shipping it at full sensitivity. A correction costs a full generation, and a correction issued at the wrong moment is wasted context. The steering paper treats the fast path as a gate for exactly this reason.

Fifth, assuming any of this transfers to domains without a verifier. MLExpert's jump to scaffold and algorithm optimization is encouraging. But windowed rewards and knowledge conditions need something to anchor to. Where outcomes can't be executed and checked, neither method has a grip.

The one thing to remember: name the decision, not the outcome

The unit of credit is the design decision, and the optimizer is downstream of it. Pick the slice of the trajectory that corresponds to a choice a human would recognize, an experiment window, a knowledge gain, a session, an intervention, and attach the reward there. Everything else in these five papers follows from that one choice.

The bottom line: what to adopt for your agent stack

If you're shipping a search agent on a fixed GPU budget, copy KnowGPO's knowledge-state definition before you copy its optimizer. An 8B model matching most 32B open-source systems at a quarter of the parameters is the largest cost delta in this cluster, and the state representation is the part that does the work.

If your production agent already compacts context or forks subagents, stop broadcasting trajectory advantages. Mosaic's session-level marginal contribution is the smallest change that reconciles training with inference, and it cuts average session count as a side effect, which is a saving you get for free on top of the accuracy gain.

If you're constrained by an executor you can't retrain, add the steering agent. It transfers to unseen executors and lets you dial intervention frequency per request without touching the base model. One thing to watch: session allocation and intervention budgeting are both being reframed as learned decisions rather than fixed hyperparameters. Expect reward functions that price tokens and wall-clock time explicitly within the next few months.