The problem: soft penalties keep failing at the edge

Reinforcement learning for safety-critical systems turns every training run into a negotiation. Task reward wants high return. The safety term wants zero violations. Put both into the same objective, as most constrained RL methods do, and the gradients push against each other. The resulting policy satisfies safety on average, which is a polite way of saying it violates constraints whenever exploration gets lucky.

Two October 2026 arXiv papers reject that compromise. FAITH (Feasibility-Aware Safety-Filtered RL) puts a learned filter between policy and environment at action execution. The unified Bellman operator paper folds safety into the value function itself, so the optimal policy is safe by construction. Different routes, same claim: hard safety constraints without surrendering task performance, even on high-dimensional systems like full humanoids.

Where the safety constraint lives

The safe-RL field splits along one question: do you enforce safety at decision time or inside the objective? That choice determines what you need to know ahead of time, what you can guarantee, and what happens when no safe action exists.

Approach	Safety guarantee	Prior knowledge required	Task performance	When no safe action exists
Constrained MDP (Lagrangian, CPO)	Only on average	None (model-free)	Competing gradient updates	Not addressed
Classical filter (CBF, reachability)	Hard, per-step	Analytic safety function + dynamics model	Myopic projection can starve long-horizon return	Undefined, projection fails
FAITH	Hard, learned and approximated	None (model-free)	Task policy trains through filtered dynamics	Falls back to minimum predicted peak harm
Joint Bellman operator	Converged policy safe at all times	None (model-free)	Maximizes constrained return once converged	Not addressed at operator level

The average-safety row is the one people keep rediscovering the hard way. Lagrangian methods work in simulation where violations are cheap. On a physical system, the episode ends the moment the constraint is broken.

FAITH: filtering after learning a safety value

Classical minimal-intervention filters pick the action closest to what the task policy proposed, subject to a safety condition, every step. They need an analytic safety function and a dynamics model to solve that projection online. FAITH replaces all of that with a learned state-action safety value, then amortizes the filtering with a feedforward network.

The neatest part is the training loop. The task policy optimizes task return through the filtered dynamics, so gradients flow through the filter. The policy learns to propose actions that will survive filtering. The safety term never appears in the task-policy update, which means no competing gradients, which is the whole point.

The "feasibility-aware" part is what separates this from prior learned filters. When every action violates the learned safety condition, a hard projection is undefined. FAITH doesn't crash or pick arbitrarily. It approaches the action with minimum predicted peak harm. That models what a safety driver actually does when a crash is unavoidable: choose the crash that hurts least. On a 29-DoF humanoid, that means the policy learns to fall away from the protected region.

A joint value function that treats safety as a hard constraint

The unified Bellman operator paper takes the opposite route. Don't separate safety from the objective. Redefine the objective so the optimal policy is safe by construction. The authors define a joint Bellman operator over a joint value function that encodes both task return and safety, then show temporal difference learning with that operator converges under a two-timescale stochastic approximation.

The fast timescale estimates the safety value of the current joint policy. The slow timescale estimates the joint value. The limiting dynamics are framed as an occupation-averaged differential inclusion that converges to the set of optimal safety-constrained task value functions. Once converged, the resulting policy maximizes task return while maintaining safety at all times along its rollouts.

That's a stronger statement than "safe on average," but it's worth being precise about what it does and doesn't promise. The guarantee applies after convergence. During training, violations can still accumulate if the safety estimate lags. The empirical results show stable convergence with near-zero test-time violations on continuous control tasks with neural approximations, which suggests the theory holds in practice, but on simpler benchmarks than FAITH's humanoid runs.

Quick Take: The two papers are complementary bets on the same problem, learn to be safe at execution time (FAITH), or learn to value safety exactly like task return (joint Bellman).

What the numbers actually say

FAITH reports the cleanest headline numbers. In Walking-Avoid on a 29-DoF humanoid: a 99.95% safety rate while retaining 97% of the unfiltered return. Translate that: roughly one unsafe step per two thousand, while keeping almost all of the task reward. That's the difference between a research demo and a policy you'd consider running on hardware.

Key numbers

• 99.95% safety rate in Walking-Avoid: about 1 unsafe sample per 2,000 steps.
• 97% of unfiltered return retained: the filter costs almost nothing in task reward.
• 29 degrees of freedom: a full humanoid, too many dimensions for hand-derived control barrier functions.
• Near-zero test-time violations with the joint Bellman operator on continuous control tasks: in practice, a converged policy passes any reasonable acceptance test.

The Push-Avoid result is the one that sold me on the feasibility-aware design. FAITH gets the highest measured safety rate by learning to sacrifice balancing and fall away from the protected region. The policy learns to give up on the task in a controlled way rather than gamble on a recovery that might not come. That's the behavior you actually want from a safety system, and you don't get it from a filter that assumes a safe action always exists.

On Safety Gym and the double integrator, FAITH achieves the highest return among methods with no feasible-start violations, and matches the lowest harm from infeasible starts. The separation matters: feasible-start safety and infeasible-start harm are different problems, and most papers only report one.

The real world: humanoids and the feasibility fallback

The same FAITH policies were demonstrated on a real Unitree G1 humanoid. That's a meaningful claim, because the feasibility fallback is precisely what makes sim-to-real viable. Hard projections assume a safe action exists. Real robots run with state estimation error, actuator lag, and contact dynamics that no analytic model captures. When the safety value says nothing is safe, the minimum-harm action is the only sane response.

For 29 DoF, you're not hand-deriving a control barrier function that covers walking, avoidance, and falling. The learning-based route isn't a convenience there, it's the only route.

The joint Bellman paper has a different practical cost: two-timescale tuning. If the fast-timescale safety estimate converges too slowly, training-time violations accumulate, and the "safe at all times" property only holds for the converged policy. That's manageable, but it's another knob to get wrong.

What trips people up

Instantaneous deviation can starve long-horizon return

Minimal-intervention filters pick the action closest to the task policy's proposal, this step. That looks safe and cheap. But minimizing only instantaneous action deviation is myopic: it can push the system into states where future rewards are unreachable, and the task policy keeps re-proposing actions that get distorted. The filter keeps you safe, and the task quietly dies. FAITH avoids this by training the task policy through the filtered dynamics, so the policy learns to propose actions that survive projection intact.

No safe action is a feature, not a corner case

Every hard-projection filter breaks when the feasible set is empty. On a physical system, that's the moment you most need a decision. I've seen projects ship a filter that throws an exception in exactly the state where a human operator needed a recommendation. Design for the infeasible case from the start, or your safety system fails at the worst possible time.

Average safety is not a guarantee

Constrained MDP methods report "low average violation rate" and everyone nods. Average violation rates hide the tail. If your policy violates once per thousand episodes but that violation is fatal, the average is a lie. The joint Bellman paper's framing, safety at all times along trajectories of the converged policy, is the right bar. Hold every method to it regardless of what its loss function says.

Feasible-start and infeasible-start metrics get conflated

FAITH reports these separately, and so should you. A filter that is great at steering a feasible state back to safe sets can be useless or harmful when the system starts already in violation. If you only report one number, you'll pay for it during deployment when the robot gets pushed into a corner and the filter has nothing to project onto.

Trusting "near-zero violations" without grading the rollout

The unified Bellman paper reports near-zero test-time violations, and that's good. But "near-zero" is an empirical claim about test distributions, not a proof about deployment. Grade the policy over long rollouts, perturbed initial states, and out-of-distribution observations before you trust it with hardware.

What practitioners hit first

Reading the discussion around both papers, the same pain points come up again and again. My team ran into the myopic filter problem on a legged locomotion task: the filter kept the robot safe, but the return collapsed because the projection kept canceling the policy's forward progress. We had assumed the filter was neutral, and it wasn't.

Others report the model requirement as the real blocker. Classical filters need a dynamics model and an analytic safety function, which rules out most contact-rich systems. The people who try to force it end up linearizing around operating points and hoping the safety proof survives contact. It usually doesn't.

The two-timescale convergence work gets the opposite reaction: impressive theory, but people want to know how to set the learning rates without a week of tuning. FAITH's answer, amortize the filter with a feedforward network and let the task policy train through it, sidesteps that tuning burden entirely, which is why it feels closer to something you could ship.

One thing to remember

A learned safety value is only as trustworthy as the data that trained it. The reason FAITH works in the real world isn't the projection accuracy, it's the fallback: when the safety value says nothing is safe, the policy still does something reasonable. That design decision matters more than benchmark percentages, because hardware always finds the state your safety function never saw.

Which approach fits which safety regime

If you're working on high-dimensional systems with no analytic model, humanoids, drones with contact, deformable objects, use a feasibility-aware learned filter like FAITH. It's the only approach in this pair that has a real answer when no safe action exists, and it's the one demonstrated on actual hardware.

If your constraints are well-defined but you need the cleanest possible guarantee, adopt the joint Bellman framing, but budget for the convergence analysis and long test-time grading. The "safe at all times" property holds for the converged policy, so your safety case rests on how carefully you validate convergence, not on the operator's elegance.

One thing to watch: hybrid designs are coming. Feasibility-aware filtering is a natural component to fold into a joint-value training loop, and within six months expect to see papers doing exactly that. If you're building a stack today, keep the filter and the value function as separate modules so you can swap in the hybrid when it lands.