RL for robots has a scaling problem

General-purpose robots need to walk across rubble and insert a connector into a socket. Sim-to-real RL is the tool that keeps showing up for both, but it has a bottleneck that compute alone can't fix. You can push parallel simulation to over a million environments, yet the learning curve barely moves. The experience in each batch is the issue, not the batch size.

This is the exact wall I hit scaling a manipulation stack: doubling environments from 16K to 256K barely moved policy returns, because most resets sampled configurations the policy either already solved or couldn't attempt yet. The reset distribution was the real constraint, and uniform sampling was flooding the buffer with useless experience.

Three open problems sit behind this wall. First, exploration in high-dimensional action spaces is brutal without guidance. Second, guidance that works is usually engineered per task. Third, the abstractions people bolt on (options, skills, subgoals) often destroy the distinctions that optimal control needs. Four papers from the October 2026 batch attack these problems from different angles, and they actually compose.

Success Guided Sampling: the data diet fix

The argument starts with a simple observation. Diverse simulator resets reduce engineering burden, but uniformly sampling over that reset distribution wastes a growing fraction of learning signal. The fraction grows because mastery is uneven: as the policy improves in one region of task space, resets there become free wins, and resets in unreachable regions produce noise.

Success Guided Sampling (SGS) fixes this by making the reset distribution adaptive. It concentrates training on task configurations near the frontier of the policy's current capabilities. That's a small change with a large consequence: when you run RL with up to 2^20 parallel environments, more than a million simulated robots at once, the sampler decides what the policy actually learns from. Massive parallelism only pays off if the loss signal per batch stays informative.

The results back this up. SGS solves multi-terrain quadruped locomotion and contact-rich assembly tasks that prior methods fail on entirely. And the transfer story is practical: the manipulation policies distill into RGB-based policies and transfer zero-shot to real hardware assembly. No real-world fine-tuning, no domain randomization tricks on top.

Quick Take: The four papers converge on a single diagnosis: exploration in robot RL is an information problem, not a compute problem, and uniform reset sampling, strict motion imitation, and unguided temporal abstraction all waste the two resources that matter, batch signal and human engineering time.

Scene graphs turn one video into a reward function

Dex-One2Many addresses the flip side: how to explore efficiently without building task-specific rewards. The setup is one human video of a tool-use task, nothing more. The trick is to abstract the video into sequential scene graphs, which act as generative constraints rather than motion templates.

This distinction matters. Motion imitation forces the policy to match demonstrated poses, and generalization collapses the moment the object starts elsewhere, the goal pose differs, or the grasp changes. Scene graphs constrain relations: object A is picked up, inserted into B, rotated to align a slot. Because relations, not coordinates, define the constraint, the simulator can sample diverse object poses and grasps beyond anything in the video. Each stage gets dense rewards from its graph node, which keeps exploration short and directed.

The 71% gap in unseen scenarios is the headline number. It means the learned policy generalizes to scene configurations the video never showed: different object placements, altered goal poses, novel grasps. And the pipeline is trained entirely in simulation, transferring zero-shot to a real multi-fingered hand across five tool-use and manipulation tasks.

Before this, when I tried strict trajectory imitation on human videos, the policy memorized the demonstrated motion and failed as soon as the scene shifted. Treating the video as a source of relational constraints instead of a motion template is the difference between a brittle demo and a generalizable skill.

Options that learn which distinctions matter

Hierarchical RL has a chronic failure mode. Option abstractions either collapse distinctions the low-level controller needs for optimal control, or keep distinctions that the high level should abstract away. Either way, hierarchy stops paying for itself.

Q-Shaped Options (QSO) sidesteps this by letting the Q functions at each hierarchy level shape the abstraction. The architecture uses distinct state-value functions, Q functions, and policies at each level, with a shared encoder forming the action abstraction between consecutive levels. The low-level Q function consumes the option as a goal, which pushes the abstraction to retain distinctions necessary for control. The high-level Q function consumes it as an action, which pushes it to discard unnecessary ones. Same encoder, two gradients pulling in complementary directions.

The practical outcome: QSO learns semantically meaningful option spaces and achieves non-zero performance in offline goal-conditioned tasks where every evaluated baseline fails. For long-horizon manipulation, that's the difference between a policy that learns a task in a week and one that learns it at all.

Here's how the four approaches compare across the dimensions that matter for deployment:

Method	What it changes	Supervision	Transfer result
SGS	Reset sampling distribution	Task configs, no demos	Distilled RGB policies, zero-shot to real assembly
Dex-One2Many	Rewards and reset constraints	One human video	Zero-shot to real multi-fingered hand, 5 tasks
QSO	Temporal abstraction	Offline trajectories	Non-zero success where baselines fail entirely
Spatial LM	State and action representation	Random play + expert demos	Higher real-robot success and target coverage

A shared vocabulary for actions and states

Spatial Language Modeling takes a representation-first stance. It encodes scene contours, goals, action targets, and future states into a shared vocabulary of discrete coordinates and semantic tokens. A task-specific grammar arranges these into spatial sequences, so one autoregressive Transformer learns action generation and action-conditioned state prediction through the same next-token objective.

The training recipe matters as much as the architecture. Random-play transitions pretrain the model, with action coordinates conditioning subsequent state predictions and excluded from the prediction loss. Then joint action and state training runs on expert demonstrations. During control, the model decodes only executable action targets and updates its history with newly observed states.

The ablation results are directly applicable. Joint action and state sequences beat training either objective alone, and random-play pretraining adds a further jump in control performance. On Push-T in simulation, the model is competitive; on the real robot, it posts higher task success and target coverage than the evaluated policy baselines. The same model, given action trajectories, predicts successive scene states, capturing the geometry of pushing. State prediction is free supervision, available at every step, with no extra labeling.

Where the four approaches fit

None of these methods competes with the others. They operate on different layers of the same stack, and you can slot them together.

The unifying pattern is guided exploration. SGS guides the data distribution to the frontier. Dex-One2Many guides rewards and resets through relational constraints. QSO guides abstraction through Q-function gradients. Spatial LM guides representation by making state prediction useful to action generation. Each one replaces a piece of hand-engineered structure with something learned.

Key numbers from this batch: 1,048,576 parallel environments: the scale SGS reaches (2^20). That's over a million simulated robots in one training run, affordable with GPU-accelerated physics, but only effective because the sampler keeps the batch informative. 71%: Dex-One2Many's success advantage over baselines in unseen configurations. Compare that with 6.5% in seen ones. Generalization, not imitation, is where the value sits. 0: real-world fine-tuning required for zero-shot transfer in both the SGS assembly and Dex-One2Many hand tasks, provided the sim trains the real control interface.

Common pitfalls

Uniform resets kill scaling. If you multiply environments without adapting the reset distribution, the wasted fraction of experience grows, and wall-clock gains flatten. Sample around the policy's capability frontier, and benchmark against uniform sampling before claiming scale helped.

Imitating the video instead of its constraints. Motion matching memorizes poses. Dex-One2Many's margin over baselines in unseen configs is the price of getting this wrong: a 71% gap versus 6.5%.

Clustering options without Q-function shaping. If your option abstraction is a state clustering or a fixed-duration primitive, you will either collapse control-critical distinctions or retain useless ones. Shape the abstraction with both the low-level goal objective and the high-level action objective, as QSO does.

Mixing the pretraining objectives. In spatial-language-style models, including action coordinates in the prediction loss during random-play pretraining teaches the model to cheat: it reads the action from the history instead of learning how actions change geometry. Exclude actions from the state prediction loss during pretraining.

Assuming zero-shot transfer is automatic. SGS and Dex-One2Many both train the real control interface in sim and distill to sensor policies before touching hardware. If the sim doesn't match the real action space and observation stream, no amount of RL fixes the gap at deployment.

One thing to remember

Guided exploration is the shared lesson. Every paper in this batch replaces a piece of brittle hand-engineering with a structure that steers learning: adaptive sampling, relational constraints, Q-function-shaped options, or a unified action-state vocabulary. If your RL training is stalling as you scale, the fix is rarely more environments. It's controlling what the agent sees next.

The bottom line

If you're scaling parallel RL for locomotion or manipulation and returns have plateaued, adopt SGS-style adaptive sampling. It concentrates experience at the capability frontier, and the paper demonstrates it solving tasks that uniform-reset baselines fail entirely.

If you have one human video per task and no robot demonstrations, go the Dex-One2Many route: extract scene graphs, sample diverse resets from relational constraints, and reward each stage. You'll get zero-shot transfer to real hardware with generalization far beyond the video's poses.

If you're building long-horizon control stacks, combine QSO-style option learning with a spatial language representation. The temporal abstraction handles horizon compression, and the shared action-state vocabulary provides free prediction supervision. Expect these two threads to merge into general-purpose policy backbones within the next few model releases.