Appearance
Your agent RL budget is going to the wrong rollouts
Training a tool-use agent with RL is an accounting problem now. The weights are the cheap part. Rollouts are not. Every episode executes real tools against real state and gets verified by something that costs money, often another model. Waste a batch and you don't get it back.
Six papers from October land on that same wall from six directions. One schedules environments. One resamples continuations that already exist in the batch. One filters out failures that shouldn't teach anything. One splits the evaluation budget across search and selection. One measures how badly teams fall apart when facts are spread across participants. One derives a scaling law for synthetic tool data.
Read together they make a single argument. Most of the wasted compute in agentic RL sits in data selection, not in the optimizer. Fix the selection and the same model, the same GPU hours, and the same benchmark budget go further.
The expensive part is the environment
An agent rollout costs roughly steps times tool calls times verifier calls. The verifier is the piece people underprice. In an executable environment you have to check whether the SQL was valid, whether the file existed, whether the API returned 200, whether the final answer matched a rubric. That runs on every episode, including the ones your policy was never going to solve.
EnvSched frames this as an environment-level data problem. Which environment-task pairs deserve the next batch, given what you've already observed? The usual answer is random sampling or a heuristic that favors tasks with mid-range pass rates. Both ignore the structure of the environment: that two tasks share a tool, that two others share a verifier, that a third pair shares a state machine.
Scheduling environments instead of tasks
EnvSched builds a dual-relation graph. Within-environment edges connect tasks that share tools, states, or verifier structure. Cross-environment edges connect environments that look similar in the abstract shape of execution and reward. Rollout statistics propagate along both edge types, and from that the scheduler predicts per-task reward variance.
The scheduling value has two halves. Exploitation picks tasks likely to produce informative reward differences. Coverage spreads the batch across environments and interaction patterns that are under-observed. Environment quotas cap how much any single environment can dominate a batch, and a similarity penalty kills near-duplicates. No pilot rollouts are needed to warm this up, which matters because pilot rollouts are the exact cost you're trying to cut.
The headline number: on Qwen3-8B, EnvSched reaches an average score of 48.81 across BFCL-v3, tau-Bench, and ACEBench-Agent after 40 training steps. Random task selection needs 120 steps to hit its best score. That is the same ballpark of quality for a third of the rollout bill, and the paper reports the gain holds across model sizes at a fixed training budget.
I've seen the failure mode this addresses. My first agent RL run added tasks whenever the curve flattened, which felt like progress and mostly bought duplicates. Half the new tasks shared a tool with something already in the training set. The gradient got noisier, the bill got bigger, and the eval plateaued.
Key numbers from the six papers
| Paper | Measurement | Value |
|---|---|---|
| EnvSched | Qwen3-8B average after 40 steps | 48.81 across BFCL-v3, tau-Bench, ACEBench-Agent |
| EnvSched | Steps that random selection needs to reach its best | 120 |
| ConCal | Gain over GiGPO on ALFWorld | +3.39 pp |
| ConCal | Gain over GiGPO on WebShop | +5.86 pp |
| GEPA-Curator | Gain over GEPA | up to 20.7% |
| GEPA-Curator | Gain over ACE | up to 21.2% |
| TeamGym | Success drop when pivotal facts are distributed | 22.9 pp |
| TeamGym | Further drop when participants must extract facts themselves | 19.9 pp |
| SynthAPI | Tasks generated and models trained in the sweep | 458,752 tasks, 849 models |
Every row is a measured delta in a controlled comparison, not a projection.
Quick take: at every layer of the stack, from which environments you roll out to which failures you learn from and which candidates you fully evaluate, the win comes from deciding what not to compute.
Recursive calibration for long-horizon credit (ConCal)
GiGPO gets credit assignment by comparing actions taken at shared states. Same state, two different actions, compare the outcomes. It works, and it has one structural weakness. Each action's value estimate rests on a single sampled continuation. If the agent flubbed something three steps later, that noise flows back into the estimate for a decision that was fine.
ConCal keeps the shared-state idea and fixes the sampling. Visits from the same task get matched by their incoming state-action-state transition. Shared continuation evidence gets combined with each visit's own transition estimate. Then the calibrated values propagate backward to refine earlier action-value estimates. No additional rollouts. The whole thing is a re-weighting over a batch you already collected.
Held-out return diagnostics show lower error in within-group value differences and better agreement with reference action orderings. On the benchmarks, with Qwen2.5-1.5B-Instruct, it beats GiGPO by 3.39 points on ALFWorld and 5.86 points on WebShop. Those numbers come from a 1.5B model, which means a single consumer card at bf16 and no cluster.
Anything that improves credit assignment for free deserves attention. The usual fix for noisy long-horizon advantage is more rollouts, which is exactly what EnvSched is trying to ration. ConCal takes the opposite route and squeezes the existing batch harder.
Some failures shouldn't produce gradients (GEPA-Curator)
Reflective prompt optimizers like GEPA read failed rollouts and rewrite the prompt that produced them. That loop assumes the failure carries signal about the prompt. It often doesn't. An agentic failure can come from a weak prompt, a scaffold that never exposed the right tool, stochastic execution in the environment, or a judge that scored a correct answer as wrong. Training on those cases teaches the optimizer to avoid tasks it could have solved.
GEPA-Curator adds two pieces to the optimizer. Task curation filters out tasks likely to yield noisy or unreliable signal. Prior success retrieval pulls past successful trajectories for a task and hands them to the optimizer as extra context, so it edits the prompt against a demonstration of the task actually working. Evaluated on GoBrowse, ALFWorld, AppWorld, and tau-bench, with both Gemini 3.1 Pro and Claude Opus 5 as optimizer models, it matches or exceeds the baselines and gains up to 20.7% over GEPA and 21.2% over ACE.
Deleting training data is the cheapest optimization available. The gains here come mostly from refusing to learn.
Spend the evaluation budget twice
Agent scaffold evolution generates candidate workflows, evaluates them, and keeps the winners. Evaluation plays two roles: it steers the search toward better candidates, and it decides which candidate you ship. Running every candidate on the full task set makes both roles expensive.
The paper's finding is that sparse feedback sustains productive evolution. You don't need a complete score for every candidate to keep the search moving. What you need is targeted evaluation at the end, because that's where the decision gets made. Replay analysis shows why: closely matched candidates can only be separated by shared informative items and a common scoring reference.
The proposed allocation framework mirrors that. A short fully-evaluated warm-up establishes a core of informative items and calibrates a response model whose item parameters stay fixed during search. After search finishes, the model gets refit, and paired successive halving reuses accumulated observations while executing only the missing evaluations. Across four benchmarks and two evolution frameworks, it beats full evaluation on mean performance in most settings using roughly half the evaluation cells.
Half the eval budget for a better mean score. If you're running scaffold search with an LLM judge, that's your bill dropping by half with no accuracy tradeoff.
Teams lose 22.9 points when facts are split (TeamGym)
TeamGym builds team decision tasks from legal, finance, medical, and software engineering corpora. Its contribution is a method for identifying pivotal information: a minimal set of facts where each element is necessary and the whole set is collectively sufficient for the correct final decision. Because the set is known, the environment can control how it's distributed and can measure whether agents found and integrated it.
Eight models on twelve source corpora produce two clean gaps. Distribute the pivotal facts across participants and success drops 22.9 percentage points against giving the same information to a single model. That's the communication gap. Force participants to identify and extract pivotal facts from source material themselves and success drops another 19.9 points. That's the retrieval gap.
The interesting result is the training one. RL on team decision loss, where the reward only scores the final decision, still produced emergent coordination: who participates, who speaks, how information moves between agents. Nobody supervised the communication. It showed up because it helped the team get the answer right.
That reframes how I'd set up a multi-agent eval. If I check a team setup by giving one agent all the context and calling it a pipeline, I'm measuring the single-agent ceiling and calling it teamwork.
Synthetic environments have their own scaling law
SynthAPI generates tool-use environments with explicit controls over diversity along three axes: tools, tasks, and the databases the tools operate on. The sweep is large enough to fit a curve: 458,752 tasks across 849 trained models of four sizes, with the generation controls varied systematically.
The resulting scaling law predicts downstream tool-use agent generalization from the scale and composition of the synthetic environments. Two lessons come out of it. Once your data budget is large, scaling tool and database diversity beats adding more tasks. And increasing task complexity buys larger gains than adding volume.
Practical translation: if your synthetic pipeline emits ten thousand near-duplicates of the same three API calls, you've scaled the wrong axis. You want a hundred tools with varied state, not a thousand phrasings of the same lookup. The agents trained under these findings beat prior work across four evaluation benchmarks at the same budget.
What trips people up
Training on every failed rollout. A failure caused by a flaky verifier or a judge misreading a correct answer teaches the optimizer to avoid a solvable task. Curate before you learn.
Scoring a task with one sample. Single-continuation value estimates carry noise from decisions made much later in the trajectory. If your rollout batch already contains shared states, use them.
Re-evaluating every candidate from scratch at selection time. Search-time scores were already paid for. Reuse the observations, refit the model, and execute only the missing cells.
Growing task count when the budget grows. Past a certain scale, tool and database diversity returns more than volume. Adding tasks that reuse existing tools buys you correlated noise.
Benchmarking a team by handing one model all the facts. That measures a single-agent ceiling, and the 22.9-point communication gap disappears from your results entirely.
One thing to remember
None of these six papers got faster by making the model better. They got faster by making the computation smaller: fewer rollouts scheduled per step, fewer failures admitted as signal, fewer evaluation cells spent per candidate. That's the cheapest lever in agentic RL right now, and it's available before you touch the architecture.
The bottom line
If you're running agent RL on a fixed rollout budget, add environment-aware scheduling before you scale the model. EnvSched's 40-step score matching random selection's 120-step best means a third of your bill is scheduling overhead, and it's the first thing to fix.
If you already store trajectories, implement ConCal-style continuation sharing. It's bookkeeping over data you have, and the 5.86-point WebShop gain comes from a 1.5B model, so you can test the idea on one GPU before committing to a training run.
If you're generating synthetic tool data or evolving scaffolds, stop adding volume. Diversify tools and databases, halve your evaluation cells with a warm-up plus successive halving, and expect the two lines of work to merge: scheduling and curation are complementary and have only been tested separately so far.