A hand that learns to catch a thrown object, a transformer that reads object positions relative to an anchor, a model that emits language and actions in a single pass, and a drone policy that has to survive a human grabbing the controls. Four recent arXiv papers, four different robots, one shared diagnosis: the policy fails because of what it's looking at, not because it's too small.

The state you hand a policy decides how much it has to learn

Standard object-centric RL gives each object a world-frame position. That's an odd thing to hand a network. Decisions depend on where things sit relative to each other: the cup relative to the gripper, the ball relative to the paddle, the traffic relative to your heading. A world-frame policy has to learn the same response separately at every location in the workspace where the same configuration can occur.

Shift the table 20 cm and the policy sees a brand new problem. It re-learns grasp approach from scratch. That's the mechanism behind the brittleness people hit when they move a camera or reconfigure a cell.

Here's how the four papers line up.

Paper	What changed	Representation	Headline result
ORBiT / ORCA	Absolute to anchor-relative coordinates	Object positions in a frame attached to a decision-relevant anchor	Aggregate HNS IQM above 1.0 across 13 Atari games, beating pixel-based PPO; Foosball learned from scratch
AdaDex	One training population to two	Shared policy over motion-tracking and generalization environments	Diverse dexterous skills from seconds of motion per skill; sim-to-sim and real-world transfer
UniWALM	Language model bolted on, to language inside	Video, action, and language co-denoised in one backbone	34.5% on the RMBench memory subset vs 28.5%; 11.1% to 61.1% on Find-the-Item
Interruption-aware RNN	Interruption-blind to interruption-conditioned recurrence	Hidden state conditioned on override context	74% fewer post-interruption collisions, 69% fewer mission failures

ORCA: attach the frame to an anchor, not to the world

ORCA stands for Object-Relative Coordinates fixed to Anchor. Entities get described in a frame attached to one anchor object instead of the global frame, and ORBiT is the transformer that consumes those anchor-relative inputs.

The gains show up in two places. Relative representations help plain MLP agents as well as transformers, which suggests the win comes from the input encoding and not from the architecture. ORBiT then clears the pixel-based PPO baseline with an aggregate human-normalized IQM above 1.0 over 13 Atari games. Past 1.0 means the agent is at or above human-level play on that aggregate, computed from object slots rather than pixels.

Foosball is the right demo to fix the idea in your head. A paddle's job depends on where the ball is relative to the paddle. Absolute table coordinates add nothing and force the policy to memorize the same strike at every row position. ORBiT learned it from scratch and dominated the baselines.

Two details carry the practical weight. ORCA extends to rotations, so heading enters the same relative frame as position. And the paper shows the gains require a decision-relevant anchor: pick a decorative object as your anchor and the advantage evaporates.

AdaDex: one clip, two kinds of practice

Dexterous hands have been stuck on a fork in the road. Goal-reaching RL generalizes to arbitrary object poses, but with no human prior it converges on finger strategies that look wrong and fail on real hardware. Motion tracking reproduces a human clip faithfully and refuses to deviate from it.

AdaDex trains one shared policy on both at once. Two populations of massively parallel environments feed it: motion-tracking environments with a dense tracking objective, which carry the human prior, and generalization environments with noised initial states and goals plus a sparse object-success reward, which force the skill to work far off the reference.

What comes out covers functional grasps, object reposing, in-hand reorientation with finger gaiting, and dynamic behaviors like throw-and-catch. The input is a few seconds of hand-object motion, from mocap or reconstructed from in-the-wild RGB-D video.

That data budget is the part worth stealing. A few seconds per skill means you don't need hours of teleoperation per behavior. You need one good clip and enough parallel simulation to noise it into a distribution.

Quick Take: these four papers mostly win by changing the observation, the training distribution, or the recurrent context, not by scaling the model.

UniWALM: language as working memory, no planner required

Most video-action models attach a vision-language model to the front and run planning as a separate stage. UniWALM puts video, action, and language inside one mixture-of-transformers backbone, with per-modality diffusion heads that denoise at the same time. Subtask text and textual memory decode in the same pass as the action. The generated language gets fed back in on the next step.

That feedback closes a language-as-memory loop with no external planner and no memory bank. Flow Matching with Correction (FMC) trains the action head to recover from perturbed trajectories, and it costs nothing at inference time.

On the memory-dependent subset of RMBench, UniWALM reaches 34.5% with no pretrained language or reasoning model. Mem-0 sits at 28.5% with keyframes, and UniWALM uses a single anchor frame. On a real-world Find-the-Item task, generating textual memory lifts success from 11.1% to 61.1%. State-of-the-art VLAs stay at or below chance there, which the paper puts at 33.3%.

34.5% is not a solved benchmark. It's a lead on a task where every method is bad, and the interesting part is where the lead comes from. Writing the memory down as text and reading it back next step does more work than adding a planner.

Interruption-aware RNNs: safe override, unsafe handback

Recurrent navigation policies integrate observation-action history, which is how they manage the safety-efficiency trade-off under partial observability. The trouble starts when something else takes control. A remote pilot, an onboard detect-and-avoid system, or a runtime assurance monitor overrides the policy for a few seconds, then hands authority back.

While the override runs, the hidden state stops matching the world. When authority returns, task variables are less decodable from that state and collision risk rises. The paper traces an action-observation feedback loop: degraded actions bend the flight path, the bent flight path produces new observations, and those observations keep the recurrent context polluted. Handback is where the risk concentrates, not the override itself.

Three architectures condition recurrent processing on interruption context: separate feature streams, dedicated interruption memory, and gated environmental features. All three preserve more decodable task information than a standard recurrent policy.

74% fewer post-interruption collisions versus the recurrent baseline. 69% fewer interruption-induced mission failures. Zero degradation in interruption-free performance for the strongest variant.

That last number is the one that decides whether this ships. A policy that's safer only when interrupted and worse the rest of the time won't survive a flight review.

What I'd try first in an existing stack

Start with the representation change from ORCA, because it's the cheapest experiment on the list. Strip world-frame coordinates out of your object-centric observation, attach them to the end effector or the target object, and rerun your evals. If your policy was quietly memorizing positions, the change shows up fast and costs you a few lines of coordinate math.

Sim-to-real on hands is where I've seen the most patience lost. The AdaDex result I care about isn't throw-and-catch. It's the noised initial states. When my own setups drifted off the demo distribution, tracking policies produced confident, human-looking motions aimed at the wrong part of the object. The fix in that paper is structural: put off-reference cases into training with a success signal attached.

For the navigation work, the cheap move is instrumentation. Log the recurrent hidden state across an override and check how much task-relevant information survives it. Most teams benchmark the override window and never look at the two seconds after.

Common Pitfalls

Anchoring to something that doesn't matter. ORBiT's advantage depends on a decision-relevant anchor. Anchor the relative frame to a static landmark and you've rebuilt absolute coordinates with extra steps.

Normalizing positions but not rotations. ORCA extends to headings. If you center your positions and leave orientations in the world frame, the policy still relearns every configuration at every angle.

Measuring safety during the override instead of after it. Handback is the risky window. Benchmark collisions in the seconds following the return of authority, or you'll certify a system on the easy half of the problem.

Running motion tracking without a generalization population. A tracking-only policy is perfect on the clip and stops dead at its edge. The second environment population with noised states and a sparse success reward is what makes the skill portable.

Bolting a planner onto a unified model. UniWALM's memory is the generated text from the previous step. Adding an external memory bank or a planning stage reintroduces the two-stage latency the design removes, and it breaks the loop that makes the memory useful. Skipping FMC to save training time is the same mistake in a different place; it costs nothing at inference.

One Thing to Remember

Every result here improves because the policy's input changed: relative coordinates instead of absolute ones, a noised training distribution instead of one clean clip, generated language instead of an external memory store, an interruption-aware hidden state instead of a blind one. Before you add parameters, check what you're feeding the network and whether it survives a shift.

The Bottom Line

If you're training object-centric policies in simulation, move to anchor-relative observations before you scale the model. It's a few lines of coordinate math, and it decides whether your policy survives a moved table.

If your dexterous skills are stuck between unreliable goal-reaching and a tracking policy bound to one clip, copy AdaDex's two-population setup: track the clip in one set of environments, reward object success in a second set with noised starts, and keep both in one policy.

If you're deploying a recurrent navigation policy with any human override or runtime assurance monitor, condition the recurrent state on the interruption context. A 74% cut in post-handback collisions and a 69% cut in mission failures, with no cost when nothing interrupts, is a cheaper engineering bill than the certification review you'll otherwise fail.