Credit assignment is the bottleneck

Reinforcement learning with verifiable rewards (RLVR) is how reasoning models get better at math. The setup looks simple: sample a long solution trace, check the final answer, return 0 or 1. But one scalar at the end of a thousand-token reply has to explain which tokens mattered. That's a credit assignment problem, and it's the thread running through most of the policy optimization papers landing this quarter.

Six preprints from this cluster hit the main pressure points. Two target RLVR credit assignment directly (BPO, HISPO). One fixes offline flow policy learning (OptiFlow). One removes environment interaction from reward design (EARS). One makes meta-RL safety-aware. One tightens the theory for submodular MDPs. Read together, they all point the same direction: the reward signal is no longer the hard part. Where and how you assign credit is.

BPO: Bellman equations without a critic

Bellman Policy Optimization (BPO) starts from a practical annoyance. Standard RLVR pipelines that use a critic need value estimates at intermediate states, and those estimates get noisy when generations are thousands of tokens long. The fix: derive the objective from Policy Mirror Descent (PMD), then use the Bellman equations to rewrite it as a trajectory-level objective. No intermediate state values needed. The paper proves the reformulation has the same unique optimal solution as the original PMD objective.

The practical loss approximates that objective. The piece that makes it work is a mismatch-correction weight, a smoothed ratio of complementary token probabilities that up-weights steps where the improved policy disagrees with the sampling policy. All without bootstrapping off a learned value function.

That matters beyond theory. A value model is another network to train, another set of hyperparameters, and another source of instability when generations run long. BPO is evidence you can skip it.

Key numbers+3.75 Acc@8: HISPO's gain over GRPO on AIME25, roughly four extra percentage points of accuracy O(H^ε): the new polynomial-time bound for submodular MDPs with horizon H, down from a ratio linear in H 0: environment interactions EARS requires; it learns from imagined trajectories alone

HISPO: the middle granularity wins

HISPO takes the other side of the same problem. Where BPO removes the critic, HISPO asks what granularity of credit assignment is correct. GRPO and DAPO apply importance-sampling correction at the token level. GSPO applies it once per sequence. Token-level treats every token equally, so a correct 50-token proof buried in 900 tokens of dead ends gets diluted. Sequence-level fixes the dilution but flattens the trajectory into a single atom, losing information about which stretch actually solved the problem.

HISPO identifies contiguous segments at rollout time using entropy, assigns soft saliency weights, and clips importance-sampling correction at the segment level. The results support the intuition. Fine-tuning Qwen3-1.7B-Base (small enough to fit on one consumer GPU) on math reasoning, HISPO improves Pass@8 over the strongest baseline on all six benchmarks, and matches or beats the strongest baseline in Acc@8 on five of six.

Correction unit	Example methods	How credit is assigned	Known failure mode
Token	GRPO, DAPO	Every token gets equal weight	Long traces dilute the signal; noisy updates
Segment	HISPO	Entropy-weighted contiguous segments	Needs rollout-time segmentation logic
Sequence	GSPO	One correction for the whole response	Flattens the trace; can't localize the solving step

The community discussion around RLVR has been split along these exact lines.

When I first ran GRPO on a long-form math task, the loss oscillated in a way that pointed at token-level importance ratios misfiring on exploratory steps that still led to the correct answer. I kept wanting a middle ground between treating every token equally and treating the whole solution as one atom. That's exactly the gap HISPO fills. People I've talked to either swear by token-level correction for its simplicity, or they've moved to sequence-level because clipped token updates felt too twitchy on long traces. The AIME25 numbers say granularity is a tunable lever, not a fixed choice.

Quick Take: segment-level credit assignment is the practical winner here. It beats token-level because long reasoning traces have long uninformative stretches, and it beats sequence-level because it can still localize the step that solved the problem.

Offline flow policies without OOD drift

OptiFlow works in a different regime: offline RL with fixed datasets that often contain multimodal action distributions. Flow policies represent multimodality naturally. Turning them into a fast one-step policy without breaking them is where things fall apart. Direct value guidance collapses modes. Maximizing a critic's estimate pulls samples into out-of-distribution regions where the critic overestimates.

OptiFlow reframes the problem as structured sample allocation. It trains a value-aware reference flow policy alongside the one-step policy, coupling their action samples through state-wise entropic optimal transport. Critic-estimated values set the priority of distillation targets; action-distance cost keeps the pairings geometrically sensible. Because the one-step policy anchors to high-value, dataset-supported modes instead of maximizing the critic, it avoids OOD divergence entirely.

If your dataset comes from human demonstrations or mixed policies, the action distribution is genuinely multimodal. A unimodal policy cannot represent two valid ways to grasp an object or merge into traffic. Transport-guided distillation is how you keep both modes.

Rewards without sampling

Reward specification is where RL deployment usually dies. Preference-based methods fix misalignment but require repeatedly training policies and sampling real trajectories. When a bad rollout means a hospital visit or a crash, that loop is off the table.

EARS (Experience-Free Autonomous Reward Specification) removes environment interaction. An LLM-mediated process builds a small set of expressive reward features from the task description and the observation space. Then the method samples imagined trajectories in that feature space and learns feature weights from preferences over the imagined pairs.

The comparison that matters: EARS beats the baseline of directly prompting an LLM to write a reward function, in three long-horizon domains (pandemic lockdown regulation, insulin dosing, highway driving), whether the preference labels come from ground truth or from an LLM. Direct prompting is the seductive shortcut, and it loses. If you're deploying RL where interaction is costly or unsafe, imagined-trajectory preference learning is the practical middle path.

Safety has to include belief

Meta-RL adapts to unseen tasks with little experience, and adaptation is exactly when an agent is most likely to break constraints. The safety framework from this cluster reasons in information space: the physical state plus the agent's belief over the underlying task. The safety value function measures the probability of avoiding unsafe regions indefinitely, satisfies a self-consistency condition and a Bellman equation, and is learnable via meta-RL. Once learned, it drives both a safety filter and constrained policy optimization.

The practical reading: an agent that believes it is on task A but is actually on task B can be unsafe in ways a state-only check never catches. Beliefs shift during adaptation, so the safety estimate must shift with them. State-only safety filters are the default, and they're the wrong default for meta-learning settings.

Submodular MDPs get a sublinear bound

The theory paper deserves attention despite the dry title. Standard MDPs assume additive rewards. Submodular MDPs generalize to monotone submodular reward functions, which capture diminishing-returns objectives like coverage and exploration. The paper gives an LP-based algorithm built on the Sherali-Adams hierarchy and a Round-or-Cut scheme.

Without the stochastic component the problem is Submodular Orienteering (find an s-t walk in a directed graph maximizing a monotone submodular function under a length constraint). Here the framework delivers an O(n^ε)-approximation in polynomial time for every ε > 0, a bound that was open even for the deterministic orienteering problem. For submodular MDPs the analogue is O(H^ε), where H is the horizon. Prior work on submodular MDPs stopped at an approximation ratio linear in H.

Linear in H is a guarantee that degrades with episode length: the longer the horizon, the weaker the bound. O(H^ε) with small ε stays meaningful as H grows. The framework also exposes a trade-off between approximation quality and how many previously visited vertices the policy conditions on, which matters for compact, memory-limited policies.

Common pitfalls

Five mistakes keep showing up in this area.

1. Using token-level credit assignment for long reasoning traces. If your RLVR task involves multi-hundred-token solutions, don't assume GRPO's token-level ratios distribute credit correctly. The signal dilutes. Try segment-level correction (HISPO) or a trajectory-level reformulation (BPO) before you add more samples.
2. Training a value network because that's what the template says. BPO shows a critic-free objective is provably equivalent to PMD. If your value model's intermediate-state estimates are noisy, delete the critic instead of tuning it.
3. Maximizing the critic directly in offline flow learning. It collapses modes and drifts into OOD regions with inflated values. Anchor the one-step policy to high-value dataset samples via transport-based pairing (OptiFlow).
4. Asking an LLM to write a reward function outright. That baseline loses to preference learning over imagined trajectories in every domain EARS tested. If you can't sample the environment, sample the feature space instead.
5. Checking safety on physical state only in meta-RL. Adaptation shifts the agent's belief over tasks. Include that belief in the safety value function, or the filter misses the cases that matter.

One thing to remember

The two RLVR papers in this batch improve results without touching the reward signal. BPO changes where credit comes from. HISPO changes the granularity at which it's assigned. If rewards were the bottleneck, the field would be publishing reward models. It isn't.

The bottom line

If you're training a reasoning model with RLVR and your generations run long, move to segment-level credit assignment. HISPO's +3.75 Acc@8 over GRPO on AIME25 shows the granularity choice buys more than most reward tweaks.

If you're shipping offline policies from fixed datasets with multimodal behavior, skip direct critic maximization and use value-weighted optimal transport to anchor your one-step policy. Critic-guided flows collapse to the wrong mode; OptiFlow keeps both.

If you're deploying RL where one bad rollout is unacceptable, invest in EARS-style preference-based reward specification without environment sampling, plus belief-aware safety filtering for adaptation. Online RLHF and state-only shields both fail precisely in the long-horizon, rare-failure regime.

Watch for GRPO alternatives becoming the default RLVR recipe within six months. The granularity axis is now a standard knob, and DAPO, GSPO, BPO, and HISPO are only the first wave.