A model that gets better at one thing gets worse at everything else

Run GRPO on a text-to-SQL task for a few hundred steps and watch pass@1 climb. Then sample the same prompt 128 times. The queries come back nearly identical, down to the table aliases. The reward went up because the sampler found a reliable path and stopped exploring.

That is sharpening, and it costs more than it looks like it does. A model that returns one correct SQL query is fine on a benchmark. A model that returns one interpretation of an ambiguous question is a support ticket. Three recent preprints attack this at three different layers of the stack. EliteReplay targets solution diversity during GRPO. SCOPED targets gradient interference between capabilities in vision-language models. A short theory note works out when the baseline terms in your policy gradient estimator help you and when they quietly bias you.

Paper	What it changes	Headline result
EliteReplay	GRPO plus a replay loss over a quality-diversity archive collected from the frozen base model	Up to 77% more distinct correct strategies than the strongest baseline, and higher pass@128 than every baseline
SCOPED	Per-capability experts, skills extracted by reward-guided reflection, on-policy distillation into one student	Beats the backbone and mixed-capability RL across eight multimodal benchmarks. Code at github.com/NEUIR/SCOPED
Baselines and score centering	Covariance formulation of the importance-weighted policy gradient	At sequence level, score centering is redundant. At token level, dropping it biases the gradient

Those identifiers are the arXiv listings as published. Both the first two are 2026 preprints, so they're recent enough that you'll want to check them against your own reproduction before you rewrite a training config.

The baseline term is doing more work than you think

GRPO implementations subtract the group mean from the reward and, depending on which repo you forked, either keep or delete score centering, that extra subtraction of the mean log-prob before multiplying by the advantage. Most people treat both as free variance reduction. They aren't free, and they aren't interchangeable.

Here is the argument. Off-policy training, whether from a training-inference mismatch or async staleness, forces importance sampling. Nobody uses the raw ratio; they truncate, mask, or clip it. Once the ratio is regularized, the importance-weighted policy gradient stops being invariant to constant shifts in reward. Add 100 to every reward and the gradient moves. That shouldn't happen, because a constant shift carries no information about which action was better.

Recovering the invariance means changing the estimand to the covariance between reward and the importance-weighted score. Group-mean baselines and score centering both center, but they subtract different means. The baseline subtracts a sample mean computed over the group. Score centering subtracts a population mean. Two unbiased covariance estimators, two different variances.

What that looks like in an update:

# Group-mean baseline, always
adv = rewards - rewards.mean(dim=1, keepdim=True)

# Token-level IS with a clipped ratio: keep score centering
lp = logprobs - logprobs.mean(dim=-1, keepdim=True)   # score centering
loss_tok = -(adv.unsqueeze(-1) * lp * ratio.clamp(0.8, 1.2)).sum()

# Sequence-level IS: score centering buys nothing once adv is centered
loss_seq = -(adv * ratio_seq * seq_logprob).sum()

Importance sampling level	Group-mean baseline alone	Group-mean baseline plus score centering
Sequence-level	Unbiased	Unbiased, no useful variance reduction
Token-level	Biased when the ratio is regularized	Unbiased

Quick Take: Score centering isn't a hygiene step you inherit from the last repo you forked. At token level with a clipped ratio it removes bias. At sequence level it's dead weight you're paying variance for.

EliteReplay: archive the strategies before training eats them

Nobody measures solution diversity well, and that's the real problem. Token entropy rises when the model varies its aliases. Two queries can differ in every token and execute to the same strategy, or differ in two tokens and represent completely different readings of the question. The paper's own complaint is that prior methods optimize metrics which don't reflect the diversity of correct solutions, and my experience agrees.

EliteReplay reframes the goal: you want a model that can produce multiple good solutions, not one reliable one. The recipe runs in two stages.

Before training, the frozen base model samples many solutions per prompt. A quality-diversity search filters them through the verifier, clusters the survivors by approach, and keeps one representative per distinct approach in an elite archive. You're snapshotting the base model's solution space while it's still wide.

During training, GRPO runs as usual and picks up an extra replay loss on the archive. The replay term keeps archived strategies from falling off the distribution.

Key numbers:

• 77% more distinct correct strategies than the strongest baseline on text-to-SQL. Diversity here counts distinct correct approaches, not token differences.
• pass@128 above every baseline, so accuracy wasn't traded away for the diversity gain.
• ~2x coverage of both interpretations on AmbiQT, an ambiguous-question benchmark. That's the number to put in front of a product team.
• 8 multimodal benchmarks, the cross-domain check that SCOPED's per-capability distillation holds up beyond one modality.

The AmbiQT result is the one I'd lead with. A question with two valid readings, and the post-trained model covers both nearly twice as often as the strongest baseline. pass@128 told you the solution set stayed wide. AmbiQT tells you the width is useful.

SCOPED: train the skills apart, then put them back together

SCOPED starts from a failure mode anyone training multimodal models has hit. Optimize several visual capabilities jointly with RL and the gradients fight each other. Improving chart reading nudges OCR the wrong way. Train one expert per capability and the interference disappears, but now you're serving five models, and each expert's knowledge is tangled up with whatever the reward model happened to prefer. That's the reward-hacking bias the paper calls out.

The recipe: train capability-specific experts, then run reward-guided reflection to extract explicit skills from each one. Skill here means a written procedural description, not weights. During on-policy distillation, those skills supervise the expert's trajectories while staying hidden from the unified student. The student has to internalize the procedure without ever seeing the cheat sheet at inference.

SCOPED beats the backbone and mixed-capability RL across eight multimodal benchmarks. The analyses are the useful part for anyone doing this at home. Skill augmentation produces more consistent expert trajectories, more attention to task-relevant visual evidence, and better retention of each expert's specialty in the merged student. Interference during joint optimization gets measured rather than assumed.

Hiding the skills from the student is the design choice I'd push back on. It works because the student can't lean on the skill text as a crutch. It also means you can't inspect what the student internalized. If someone on your team needs to explain why a model attended to a particular region of an image, this architecture makes that harder.

What you'll hit when you actually run this

My own experience with GRPO-style post-training lines up with these papers' premises more than their reported numbers. Two things bit me.

The collapse is fast. On a SQL agent, pass@1 kept climbing for two days while the set of distinct query shapes the sampler produced had already flattened. Had I tracked distinct execution plans instead of reward, I'd have caught it on day one. That's the whole argument for building the archive before training starts rather than after you notice the problem.

Verifier design decides what diversity even means. If your verifier only checks the final answer, two different reasoning paths that land on the same string look identical to the training loop, and the replay loss has nothing to preserve. If it checks intermediate structure, you get a real strategy signal and you've hardcoded your definition of a strategy into the data pipeline.

Heterogeneity between experts is a feature, and teams keep trying to remove it. In SCOPED, the experts are supposed to diverge. Distill them into one student only after the skills are explicit. Merge the experts first and you get the original gradient conflict back, plus reward-hacking artifacts baked into the merged weights.

Common pitfalls

1. Measuring diversity with token entropy. Token-level variation and strategy-level variation aren't the same thing. A model that renames a table alias ten ways looks diverse and behaves identically. Cluster by execution result or by a strategy signature before you trust any diversity number.

2. Stacking score centering and a group-mean baseline without checking the importance sampling level. At sequence level, score centering adds variance for nothing once the group mean is subtracted. At token level with a clipped ratio, dropping it biases the gradient. The sampling level decides, not your config file's family resemblance to the repo you forked.

3. Building the elite archive after training. EliteReplay collects from the frozen base model for a reason. Archive post-RL solutions and you're archiving the collapse you were trying to prevent.

4. Reusing stale rollouts without recomputing the importance ratio. Async training is where staleness lives, and staleness is exactly what forces regularized importance sampling, which is what breaks reward shift invariance to begin with. The two problems compound.

5. Merging capability experts before extracting skills. The explicit skills are the transferable artifact. Fuse the experts first and the student inherits each expert's reward-hacking biases along with its capability.

The bottom line

If your post-trained model produces one solution shape per prompt, build the elite archive from the frozen base model before you start training, then add the replay loss. It costs one sampling pass over your prompt set and it drops into an existing GRPO stack without new infrastructure.

If accuracy is fine but users complain about ambiguous inputs, stop tuning entropy and measure coverage of distinct correct interpretations instead. EliteReplay got its roughly 2x AmbiQT coverage by optimizing exactly that, and pass@128 went up at the same time, so you aren't choosing between diversity and accuracy.

If you run token-level importance sampling with a clipped ratio, keep score centering. If you run sequence-level, delete it and stop paying the variance. And if you're training one VLM on chart reading, OCR, and spatial reasoning simultaneously, expect measurable gradient conflict: separate experts, explicit skills, on-policy distillation is the path that scales without a per-capability serving fleet.