Where Embodied Priors Come From When Robots Are Expensive

The most expensive thing in robot learning is a robot. Every hour of manipulation data costs a facility, an operator, a rig, and a repair budget. So the field keeps circling one question: how much of what a robot needs to know could be learned somewhere else?

Two recent arXiv papers attack that question from opposite ends. One asks whether a model can pick up useful control priors from data that has no body at all. The other asks what it takes for a vision-language-action model to sit down at a computer and play an open-world game through the same keyboard and mouse a human uses.

They don't cite each other. They don't need to. Read side by side, they map the two halves of the same problem: where priors come from, and what happens once those priors have to survive a long task.

Paper one: pretrain on systems that cannot move

"Learning Robot Behavior Priors from Non-Robotic Synthetic Data" (arXiv 2610.31275) starts from a blunt observation. Real robot data is scarce across tasks and morphologies, and embodied foundation models that promise zero-shot physical transfer are stuck behind that scarcity.

The authors ask a narrower, testable version of the question: before a model has ever seen a real robot behave, how much structure can it absorb from systems with no embodiment?

Their setup is robot action prediction. That's a sensible pretraining objective because it forces the model to commit to a specific action given a state, which is exactly the thing you want to transfer. They build three data generation pipelines that encode mathematical abstractions of robotic control, then pretrain a Transformer on the output.

The result that matters: a model trained purely on this non-robotic data gets non-trivial zero-shot prediction performance on unseen physical robot tasks, through in-context learning. No fine-tuning on the target robot. The model sees a few examples in context and produces usable actions.

The second result is the one I'd underline. The specialized control priors beat generic causal priors. In other words, the abstraction you choose to generate data from is doing real work. Feeding the model arbitrary synthetic sequences with causal structure is worse than feeding it sequences that already look like control problems. The prior has to be shaped, not just large.

Then there's the honest caveat. A compute-matched baseline trained on real robot data outperforms the synthetic-pretrained model, but only on the datasets it was trained on. That qualifier is doing a lot of work. The real-data model wins in distribution and loses whatever edge it has once you leave the datasets it saw. The synthetic prior buys breadth at the cost of peak in-distribution accuracy. For a foundation model, that's usually the trade you want.

The two papers on the same axes

	Synthetic priors (arXiv 2610.31275)	HiSTEVE 1.0 (arXiv 2610.31162)
Pretraining signal	Procedurally generated, non-embodied control data	Minecraft gameplay via imitation, RL, world knowledge injection
Model	Transformer trained on robot action prediction	Single VLA inside a hierarchical agent harness
Embodiment during training	None	Native keyboard and mouse in a game
Evaluation	Zero-shot transfer to unseen physical robot tasks	MineStudio Bench, OpenHA, long-horizon crafting
Headline number	Non-trivial zero-shot accuracy, no fine-tuning	85.0% atomic task success, 37.4% OpenHA average
Main limit	Real-data baseline still wins in distribution	Long-horizon success is under half the atomic rate

The two rows at the bottom are the whole story. Both papers get a strong number on the easy axis and pay for it on the hard one.

Quick Take: Non-embodied synthetic data is a real source of control priors, and hierarchy is what makes long-horizon VLA control possible at all, but neither paper closes the gap to a robot that just works.

Paper two: a VLA that uses your keyboard

"HiSTEVE 1.0: Harnessing Human-Interface VLA Agents for Open-World Games" (arXiv 2610.31162) picks the opposite research strategy. Instead of avoiding embodiment, it commits to the messiest embodiment available: Minecraft, with vision-only perception and native keyboard-and-mouse actions.

The problem it names is precise. Existing Minecraft agents give you one of two things. Either native visuomotor control with no long-horizon autonomy, or high-level planning that executes through environment APIs and predefined skills. The first is embodied but short-sighted. The second plans well but cheats, because it never touches the interface a human touches.

HiSTEVE is a hierarchical VLA with a capability-aware orchestration harness. The orchestration level holds structured memory for progress tracking and a VLM planner that selects subtasks, constrained to what the VLA can actually do. The control level is a single VLA that executes every subtask with vision-only input and native keyboard and mouse.

Two design details are worth stealing.

The first is the policy-coupled completion detector. Instead of deciding "is this subtask done" from pixels alone, it conditions on the VLA policy's internal context to propose candidate completions. That matters because the failure mode of every hierarchical agent I've seen is the same one: the planner thinks the subtask finished, moves on, and the whole plan silently desynchronizes from the world.

The second is the normalization split. GUI clicks are discrete, mouse movements are continuous deltas, and they live in wildly different token distributions. Sharing normalization statistics across both regimes lets one drown out the other. HiSTEVE uses regime-specific RMSNorm branches plus separate mouse-movement token embeddings to keep them apart. It's a small fix that costs almost nothing at inference and prevents a specific, ugly class of interference.

Training runs through three stages: world knowledge injection, imitation learning, and reinforcement learning with environment feedback. The RL stage is there for recovery, which is what you'd expect from a system that has to survive its own mistakes over hundreds of steps.

Why 85.0% atomic turns into 37.4% long-horizon

An 85.0% atomic success rate means the VLA gets roughly six of every seven individual subtasks right. That's a strong policy. A 37.4% average on OpenHA means fewer than two in five long-horizon tasks complete.

The gap isn't a mystery, and the arithmetic is unforgiving. Chain six subtasks at 85% each and you get 0.85⁶, which is about 37.7%. That number lands almost exactly on the reported 37.4%. So HiSTEVE is behaving close to the theoretical ceiling of a per-step success rate applied to a six-step chain. The policy isn't degrading at the end of long tasks. The tasks just compound.

What that means practically: at 85% per step, a twenty-step task (craft an iron pickaxe from scratch is more than twenty steps) finishes somewhere near 3% of the time. The paper reports completion of exactly that task, which tells you the harness is recovering better than the naive product, or the effective chain is shorter than it looks. Either way, the ceiling is the per-step number. Push atomic accuracy from 85% to 95% and a ten-step chain goes from 20% to 60%. That's a bigger return than any planner improvement.

What I ran into building agents like this

I've spent time wiring language-model planners onto visuomotor policies for game environments, and HiSTEVE's design choices read like a response to bugs I've personally filed.

The completion detector is the one that got me. My first hierarchical setup decided subtask completion from a screenshot classifier, and it drifted constantly. The agent would finish mining three logs, the classifier would fire on two, and the planner would build a crafting table it didn't have materials for. Then it would loop: try to craft, fail, re-plan to gather wood, decide it already had wood, try to craft again. It burned the entire episode budget in that loop and produced nothing. Conditioning completion on the policy's own state, as HiSTEVE does, is the fix I should have reached for first.

The capability-aware planner is the other one. Giving a VLM free rein to pick subtasks sounds powerful until you watch it ask a policy to perform an action the policy has never seen. My agents would confidently propose "trade with villager" when the VLA had only ever learned to mine and place blocks. The result is a policy doing something arbitrary and the planner recording failure. Constraining the planner to the VLA's actual skill set sounds like it would make the agent dumber. It makes it finish tasks.

My team also hit the GUI versus non-GUI interference problem without naming it. Early on, our click accuracy was fine and our mouse-delta accuracy was terrible, or the reverse, never both. Splitting the normalization per regime is the kind of thing you find after two weeks of staring at per-token loss curves. Seeing it as an explicit architectural branch with separate mouse-movement embeddings was genuinely satisfying.

Key Numbers

• 85.0% atomic task success on MineStudio Bench, which is roughly six of every seven subtasks.
• 37.4% average long-horizon success on OpenHA, close to 0.85⁶ = 37.7%, meaning compounding dominates the loss.
• 3 synthetic data pipelines in the priors paper, all built on mathematical abstractions of robotic control.
• 0 real robot hours used to pretrain the model in arXiv 2610.31275.
• 3 training stages for HiSTEVE: world knowledge injection, imitation learning, RL with environment feedback.

Common Pitfalls

Treating zero-shot priors as a replacement for real data. The priors paper is explicit that a compute-matched real-robot baseline still wins on the datasets it trained on. If your deployment is narrow and you have in-distribution robot data, collect it. The synthetic prior is for breadth and for the cold-start case where you have nothing.

Generating synthetic data without shaping the abstraction. Generic causal priors underperformed the control-specific ones in the same experiments. Pasting "synthetic data" into a training pipeline and expecting transfer is the wrong read. What transfers is the structure you deliberately encoded, not the volume.

Sharing normalization across heterogeneous action regimes. If your action space mixes discrete events (clicks, key presses) with continuous streams (mouse deltas), one will dominate the shared statistics. HiSTEVE splits RMSNorm by regime and gives mouse movements their own token embeddings. Do the same, or at minimum check per-token loss by action type before you blame the policy.

Giving the planner no knowledge of the policy's capability boundary. An unconstrained VLM planner will cheerfully propose subtasks the VLA cannot execute, and the episode ends in a failure that looks like a control bug. Restrict subtask selection to the learned skill set. Your agent will look less ambitious and finish far more tasks.

Judging subtask completion from perception alone. Pixel-based completion detectors drift within a few subtasks, and once the planner's belief state desynchronizes from the world, the rest of the episode is wasted. Condition the detector on policy context, and add a cheap consistency check against structured memory.

Measuring only atomic success. An 85% atomic number sounds great and hides a 37% long-horizon reality. If you report one metric, report the long-horizon one, because that's the number a user experiences.

One Thing to Remember

The two papers are answering the same question at different layers. One shows that the priors do not have to come from a body. The other shows that once you have a body, the binding constraint moves from "can the model act" to "can the system hold a plan together across many actions." Neither problem substitutes for the other, and the second one has a hard, arithmetic ceiling set by the first.

The Bottom Line

If you're starting an embodied project with no robot data budget, pretrain on procedurally generated control abstractions before you touch real hardware. The zero-shot action prediction in arXiv 2610.31275 is a genuine cold-start win, and control-specific priors beat generic causal ones, so spend the effort on the abstraction rather than the dataset size.

If you already have a VLA policy that works on short horizons and you're building the orchestration layer, invest in completion detection and planner constraint, not a bigger planner. At 85% atomic accuracy the long-horizon ceiling is arithmetic, not architectural, so lifting atomic accuracy by ten points buys more than any harness rewrite.

If you're evaluating claims in this space, watch the qualifier after the win. The real-data baseline's advantage in the priors paper and HiSTEVE's 37.4% long-horizon average are the two numbers that tell you what these systems still can't do, and both are stated plainly in the papers.

References

• Learning Robot Behavior Priors from Non-Robotic Synthetic Data, arXiv:2610.31275, http://arxiv.org/abs/2610.31275v1
• HiSTEVE 1.0: Harnessing Human-Interface VLA Agents for Open-World Games, arXiv:2610.31162, http://arxiv.org/abs/2610.31162v1