Appearance

Four Ways to Give a Language Model Spatial Sense

October 15, 2026 14 min read
#3d-reasoning #spatial-intelligence #multimodal-llm #novel-view-synthesis #3d-composition

Four Ways to Give a Language Model Spatial Sense ​

Everyone building a geometry-aware multimodal model pays the same tax. You bolt a 3D encoder onto the side of the LLM, run it on every query, and eat the latency. GeoThinker-F deletes that encoder from the inference path and comes out 3.6x faster, cutting per-query latency from 1.45 seconds to 0.40 seconds. That's the difference between a tool that feels responsive and one that feels like a batch job.

The other three papers in this cluster attack the same missing piece from different directions. Ask4NV asks whether teaching a model to answer spatial questions also makes it better at rendering new viewpoints. ReferFlow argues that grounding a referring expression is a path, not a point. PhySGA stops treating geometry as a static prior and drops the model into a physics simulator.

None of the four papers try to make the external 3D encoder faster. They try to remove the need for it.

The tax everyone pays for spatial reasoning ​

A geometry-aware MLLM today looks like this: a vision encoder feeds tokens to a language backbone, and separately a 3D encoder (a point-cloud network, a depth estimator, a NeRF-derived feature extractor) adds structural priors. The language model gets both streams and reasons over them.

That second stream is expensive. It runs on every forward pass, it needs its own weights in memory, and it doesn't share computation with the language stack. On a paper benchmark the overhead is a footnote. In production it's the reason your spatial QA service can't hold a conversation.

Key Numbers3.6x inference speedup for GeoThinker-F over external-encoder baselines 1.45 s → 0.40 s latency per spatial query, fast enough for interactive use 2 training stages before the external 3D expert can be removed 4 trajectory types in ReferFlow-Bench: Refinement, Shift, Reset, Mixture 4 geometry-constrained refinement rounds in ReferFlow's parallel coordinate decoding

GeoThinker-F: predict the geometry instead of encoding it ​

The core claim of GeoThinker-F is that intermediate hidden states in a causal MLLM already carry enough signal to predict current geometry and near-future semantics. You don't need an exact reproduction of a dedicated 3D encoder's features. You need something close enough to be useful.

Training runs in two stages. In stage one, the MLLM backbone stays frozen while lightweight Latent Frame Prediction (LFP) heads learn from an offline 3D geometry expert plus next-frame semantic targets. In stage two, the expert is gone. The model generates its own priors and selectively integrates them through a Spatial-Grounded Temporal (SGT) Fusion module.

The 3.6x speedup is the headline, but the more interesting result is that the model keeps competitive spatial reasoning scores across benchmarks after the expert is deleted. That tells you the geometry signal in the hidden state was always there. The external encoder was mostly doing the job of a decoder head.

Quick Take: All four papers face the same question about where geometry should come from, and each one answers it differently: the model's own latent state, a QA objective, an explicit trajectory, or a simulator.

Ask4NV: spatial question answering as free supervision ​

Ask4NV starts from a gap in existing novel-view synthesis. Recent methods lean hard on fine-grained geometric cues like depth and correspondence, but they skip high-level semantics. Object identity, attributes, inter-object relationships. A renderer that knows where every pixel goes but not that the red mug is left of the laptop will produce a plausible image of the wrong room.

The architecture is a pretrained unified multimodal model with a Mixture-of-Transformers (MoT) backbone. Camera geometry gets injected into both the VAE and the ViT representations, and understanding features flow into generation through joint self-attention with a modality-aware attention masking strategy.

The ablation that matters: joint training improves cross-view correspondence in the VAE representations and increases similarity in a 3D-aware feature space. So answering "which object is closer to the window" during training changes what the generative latents look like. The supervision isn't a side task. It reshapes the representation the renderer draws from.

That has a practical consequence. If you're fine-tuning a view synthesis model and you have any spatial QA data lying around, it's not dead weight. It's a regularizer on your latents.

ReferFlow: grounding is a trajectory, not a point ​

Most referring expression comprehension benchmarks score one thing: did the model put the final box in the right place. That's a weak test. A query like this one breaks current models:

"the mug to the left of the laptop that's on the desk by the window."

To land that box you first find the window, then the desk, then the laptop, then the mug. Each step depends on the previous one, and a model can get the final box right for the wrong reason, or stall on step two and never recover.

ReferFlow formalizes this as trajectory-level REC. ReferFlow-Bench labels each query with an ordered sequence of grounding steps and classifies the region transitions into four types: Refinement, Shift, Reset, and Mixture. Current models struggle to hold intermediate references steady across successive steps.

The fix has two parts. Stateful Region Latent Reasoning represents each target region as an object-indexed state made of continuous latents, with explicit visual and spatial supervision so region information survives into later steps. Trajectory-Aware Parallel Decoding then recovers all coordinate blocks in parallel across four geometry-constrained refinement rounds, each block conditioned on its trajectory prefix. This kills within-box sequential dependencies and eliminates cross-box autoregression, which is where a lot of the error compounds. Grounding-Aware GRPO then trains the model to stay consistent across steps.

If you've shipped a document or UI grounding system, you know this failure mode. The model nails the final bounding box 80% of the time and the intermediate reasoning is nonsense half of that. Trajectory-level eval is the metric that catches it.

PhySGA: when the simulator is the teacher ​

3D composition means placing objects in a scene so the result holds up. Previous methods either have an LLM predict coordinates directly or use diffusion priors to synthesize an image for visual alignment. Both rely on static priors in digital space, and both fall apart on tasks that need contact.

Precise hanging. Gravity-driven settling. Hidden containment, where an object ends up inside another one.

PhySGA runs composition as an interactive process inside a simulator. An Assembly Agent handles geometric and kinematic reasoning for global step planning and coarse placement. A Critic Agent reasons over contact dynamics, diagnosing simulator state and driving fine-grained closed-loop correction.

Action Routers turn semantic plans into executable physical primitives. A Self-Evolving Skill Library distills trial-and-error feedback into reusable experience, which cuts the number of exploration attempts a new task needs. Experiments in Isaac Sim report gains over existing methods on relation fidelity, physical plausibility, and visual quality.

The honest limitation: this is a closed-loop system that needs a simulator. You can't run PhySGA on a static image. But for robot simulation and digital content pipelines, where a simulator already exists, the trial-and-error loop is the whole point. Coordinates predicted by an LLM are a guess with no feedback. Coordinates refined against contact dynamics are a solution.

How the four compare ​

PaperCore problemGeometry source at inferenceHeadline result
GeoThinker-FExternal 3D encoders add runtime costSelf-predicted latent world representations3.6x faster, 1.45 s down to 0.40 s
Ask4NVNovel-view generation ignores object identity and relationsJoint MoT attention over understanding tokensBetter cross-view correspondence in VAE latents
ReferFlowREC benchmarks score only final targetsTrajectory-aware parallel coordinate decodingLeads general REC and ReferFlow-Bench
PhySGAStatic priors fail on contact-rich compositionSimulator contact dynamics plus skill libraryBetter relation fidelity and physical plausibility in Isaac Sim

Read the middle column again. GeoThinker-F and Ask4NV both move the geometry signal inside the model, one through prediction heads, one through the attention mask. ReferFlow and PhySGA keep the signal external, but make it structured. A trajectory prefix in one case, simulator state in the other. The split is real: it's the difference between distilling geometry into weights and querying geometry at runtime.

What trips people up ​

  • Don't assume GeoThinker-F removes the 3D expert entirely. Stage one still trains LFP heads against an offline geometry expert. You need that expert at training time, plus the next-frame semantic targets. Teams that skip stage one and jump straight to joint optimization lose the signal the whole method depends on.
  • Don't keep scoring REC on final targets only. In the trajectory setting, a model can hit the final box while getting every intermediate reference wrong. If you deploy a grounding system and only log final-box accuracy, you'll miss the failure mode ReferFlow-Bench was built to expose.
  • Don't generate coordinate blocks autoregressively. ReferFlow's parallel decoding across four refinement rounds exists precisely because cross-box autoregression compounds error. If you serialize box N+1 on box N, expect the last box in a five-step query to be the worst one.
  • Don't hand an LLM raw coordinates for contact-rich composition. Coordinated prediction works for coarse layouts and fails on hanging, settling, and containment. Those need a critic reading contact dynamics, or a skill library that remembers what worked.
  • Don't drop camera geometry from the VAE branch. Ask4NV injects it into both the VAE and ViT representations. If you only feed poses to the understanding tower, the generative latents have no anchor for view consistency.

One thing to remember ​

The external 3D encoder was never the point. It was scaffolding for a geometry signal that a sufficiently trained language model can predict, question, or query on its own, and all four papers are pulling pieces of that scaffolding down.

The bottom line ​

If you're shipping an interactive spatial assistant and latency is your blocker, GeoThinker-F's training recipe is the one to copy. Two stages, a frozen backbone while the prediction heads learn, then joint optimization with the expert removed. You get a 3.6x speedup and keep the spatial scores.

If your problem is grounding accuracy on multi-step queries, skip the final-box metric and build the trajectory eval first. ReferFlow-Bench's four transition types (Refinement, Shift, Reset, Mixture) give you a way to see where your model actually stalls, and parallel coordinate decoding is the fix that follows.

If your pipeline involves physical contact, assembly, or a robot simulator, don't ask an LLM to predict coordinates. PhySGA's assembly-plus-critic loop is slower per step and correct in ways a coordinate regression never will be. And watch the training-time cost of these methods over the next six months: removing inference overhead usually means moving it into a two-stage fine-tune, and somebody is going to productize that recipe.

References ​

Support Me