Appearance
Metric, Editable, or Both: Three Papers Attack the 3D Consistency Gap
A 200-photo scan of a room gives you geometry you can measure but can't edit. A text-to-3D model gives you geometry you can edit but can't measure. Getting both at once, from the same pipeline, is still an open problem, and three recent arXiv papers attack it from three different ends.
Three papers, one shared headache
The common thread is that "3D reconstruction" has quietly become three separable problems. Recovering structure from images is one. Knowing how big that structure is in real units is another. Letting a human change it without breaking the other two is a third.
Most pipelines handle two of the three well and hand-wave the rest. Here's the split:
| Paper | Problem it solves | Key idea | What it still needs |
|---|---|---|---|
| VidEdit3D | Edit a Gaussian scene without breaking multi-view consistency | Treat camera motion as video time, borrow a video diffusion prior | A rendered trajectory and a VDM pass per clip |
| GenSfM | Recover object shape and camera poses from 4 uncalibrated views | Jointly denoise shape and pose latents in one diffusion transformer | Object-centric scenes, no known intrinsics required |
| Insta360 IMU scale | Recover metric scale without a phone | Linear visual-inertial alignment over SfM poses, per-segment | A 1 kHz IMU in the camera, dual-fisheye video |
The interesting part isn't any single result. It's that all three papers assume the other two problems are somebody else's job, and each one gets unusually specific about the handoff.
VidEdit3D: editing a scene by flying a camera through it
Editing a Gaussian splat is easy to describe and hard to do. You want to change one chair. You don't want the wallpaper to shift, the lighting to drift, or the view from behind the sofa to stop matching the view from the front.
The existing options each lose something. Image-based editors apply a 2D edit to rendered frames one at a time, which looks great per frame and disagrees across frames. Methods that lock the geometry keep views consistent but refuse structural changes, so "turn this chair into a stool" comes out as a texture swap. And a single anchor view is a weak reference when the object is seen from an angle the editor never anticipated.
VidEdit3D's move is a reframing: a camera moving through a static 3D scene is the same thing as time passing in a video. If that's true, then every spatiotemporal prior baked into a video diffusion model applies to 3D scene editing for free.
The pipeline has four parts, and only one of them touches the Gaussians.
Trajectory-Guided Rendering picks the informative keyframes and cuts smooth clips that match what the video model expects to receive. Flow-Aligned Video Editing is training-free and carries the edit from the anchor view into those clips. Inside it, Editing Consistency Guidance contrasts the source and edited image conditions at a shared latent state, with scheduled source-image amplification that pulls the result toward the edited reference while holding onto the original scene context. Then Region-Selective 3DGS Refinement confines geometric and appearance updates to the target primitives, which is the step that stops background attributes from leaking into the edit.
That last piece matters more than it sounds. Most Gaussian editing failures I've seen aren't failures of the edit itself. They're failures of containment, where changing a lamp post also nudges the pavement behind it by a few centimeters.
Quick take: VidEdit3D's real contribution is the reframing, not any single component. Once camera motion is time, every consistency trick from video diffusion transfers to 3D editing unchanged.
Where trajectory-shaped editing gets expensive
The trade-off is compute, and the paper is upfront that the editing stage is training-free rather than cheap. Every clip you render along the trajectory is a video diffusion pass. A full survey with a dozen viewpoints means a dozen forward passes through a model that was never designed for real-time use.
Two failure modes are worth flagging before you build on this.
Fast camera motion relative to the video model's temporal window compresses too much parallax into too few frames, and the spatiotemporal prior stops helping. Slow, over-long trajectories blow past the window in the other direction, and the clip gets truncated mid-edit. The keyframe selector is doing more load-bearing work than the paper's framing suggests.
The other one is topological edits. Change a chair into a stool and the silhouette changes, which means the Gaussians under the old silhouette have to move or die. Region-selective refinement restricts where updates land, but it doesn't tell you what the correct new geometry is. Structural edits still lean hard on the video model's imagination, and that's where multi-view consistency is thinnest.
GenSfM: four photos, and the parts you can't see
Classical structure from motion has an honest limitation: it recovers geometry that's consistent with the input images and nothing else. Point a four-view rig at an object and you get a shell with holes where the camera never looked. Native 3D generation has the mirror-image problem. It produces plausible objects that are related to your input images but not necessarily faithful to them.
The obvious fix, running reconstruction and then generation in sequence, lets errors cascade. A slightly wrong pose from stage one becomes a badly placed surface in stage two, and there's no mechanism to correct it because the two stages never talk.
GenSfM puts them in the same denoising loop. A multi-modal diffusion transformer jointly denoises a shape latent and per-view pose latents under shared self-attention, so the geometry gets grounded in the observed views while object-level priors fill in the unobserved regions. Poses and shape are solved together rather than one after the other.
The tokenization choice is what makes the joint sequence tractable. A compact 3D-native UV-volume representation keeps the sequence small as you add views, so a fifth and sixth view don't blow the attention budget. A final pose-aware stage refines geometry and texture by sampling per-voxel image features at the recovered camera poses.
The comparison against a strong native-3D generation baseline, at four uncalibrated views:
| Property | Classical SfM | Native 3D generation | GenSfM |
|---|---|---|---|
| Geometry from observed views | Faithful | Approximate | Faithful, grounded |
| Unseen regions | Missing | Plausible | Plausible and grounded |
| Input-view poses | Recovered | Not directly | Recovered, over 96% Acc@5° |
| Known intrinsics required | Usually | No | No |
| Error cascade between stages | Not applicable | Not applicable | Avoided by joint denoising |
On Toys4k and GSO at four views, GenSfM improves novel-view PSNR by 3.2 to 4.5 dB over that baseline. Three dB is roughly the gap between a render you'd wince at and one you'd put in a slide deck, so this isn't a rounding-error improvement.
Median Chamfer distance drops by 2.8 to 3.8 times. In plain terms, the worst surface errors shrink to about a third of what they were, which is the difference between an object that looks right from the front and one that survives a turntable.
The pose numbers are the part I'd flag to anyone building a scanning app. Over 96% Acc@5° on input-view poses, with no external pose-estimation backbone, no post-hoc bundle adjustment, and no known intrinsics. Five degrees of pose error at a two-meter working distance is roughly 17 cm of lateral drift, so it's accurate enough to seed a refinement stage but not accurate enough to skip one.
The scale problem, and a 1 kHz IMU
Structure from motion from dual-fisheye video recovers geometry only up to a similarity transform. Scale is a free parameter. The standard fix is to capture ARKit or ARCore poses from a phone alongside the camera and use those to anchor real-world units, which works and is annoying: a second device, a second clock, and a second set of drift characteristics.
The Insta360 X5 and X6 have an IMU inside, recorded at 1 kHz. This paper asks whether that's enough on its own. The authors characterize the sensor first (noise density, camera-IMU synchronization, accelerometer scale factor, bias stability), estimate camera-IMU rotation and frame timing from the data, then recover scale with a linear visual-inertial alignment over the SfM poses of an entire survey.
The naive formulation has a specific bug. Share gravity and accelerometer bias across a whole survey and the scale comes out 3.4% too high on average. On a 50-meter survey that's a 1.7-meter systematic overshoot, which is the kind of error that looks fine on a screen and fails a site inspection.
Key numbers
- 0.74% mean absolute scale error against a co-mounted LiDAR scanner across nine surveys, versus 1.94% for the smartphone AR scale
- 3.4% average scale overestimate from the standard global alignment formulation
- 1.28% standard deviation, and -0.07% mean error, across 30 public benchmark sequences rescaled to 1/40, 1, and 40
- 2.6% standard deviation between inertial and smartphone scale on 50 production surveys, with the inertial estimate 0.9% lower on average
- 1 kHz IMU sample rate, no second device required
Estimating each contiguous segment independently and then averaging removes the bias. On printed fiducial markers, both the inertial and smartphone scales land inside the 1% uncertainty of the target, so the two agree where you can check them.
| Setup | Reference | Result |
|---|---|---|
| Nine surveys, co-mounted LiDAR | LiDAR | 0.74% MAE (inertial) vs 1.94% (smartphone) |
| Printed fiducial markers | Marker target | Within the target's 1% uncertainty |
| 30 benchmark sequences, scales 1/40, 1, 40 | Ground truth | -0.07% mean, 1.28% std |
| 50 production surveys | Smartphone AR only | 2.6% std, inertial 0.9% lower on average |
| Naive global alignment | LiDAR | +3.4% overestimate |
The honest caveat is in the fifty production surveys, where the phone is the only available reference. The two scales agree to a 2.6% standard deviation, and the difference grows with the spatial extent of the smartphone anchor. That means the phone reference is itself drifting, and the paper can't attribute the gap to either side.
Where the numbers actually land
Pulling the error budget together across the three claims makes the trade-offs visible. The chart below puts four reported figures on one axis. They come from different experiments and aren't directly comparable, but they show where the error actually lives.
On a 20-meter room survey, 0.74% is about 15 cm and 1.94% is about 39 cm. That's the difference between a model you can drop into a CAD overlay and one you have to re-register by hand.
The last time my team ran a 360 survey, the phone-anchor step is exactly what bit us. The phone drifted in a way that was invisible until we tried to align two scans of the same hallway, and the discrepancy grew with the size of the area the phone had walked. Segmenting the inertial alignment and averaging is a small change with an outsized effect on that failure mode.
The three papers stack into a coherent story. GenSfM solves poses and geometry together so pose error doesn't poison shape. Insta360's IMU alignment pins the whole reconstruction to real units without a second device. VidEdit3D then lets you change things without unpicking either. Nothing here is a full pipeline, but the interfaces are getting clean enough to bolt together.
What trips people up
Running global visual-inertial alignment across a long survey. The shared-gravity, shared-bias assumption biases scale upward by 3.4% on average. Estimate each contiguous segment independently, then average. It costs you an afternoon and buys back meters.
Treating ARKit or ARCore poses as ground truth. They drift, and the drift grows with the extent of the anchor. On the production surveys here, the inertial and phone scales differed by 2.6% one sigma, and the paper can't tell you which one is wrong. If you have a LiDAR scan or a fiducial target, use it.
Concatenating reconstruction and generation. Running SfM and then a 3D generator lets stage-one pose errors become stage-two geometry errors with no path back. If you need both, the two have to share a denoising or optimization loop.
Assuming a 360 sweep is a valid editing trajectory. VidEdit3D expects smooth clips inside the video model's temporal window. Fast pans compress parallax, slow orbits get truncated, and both push the edit outside the region where the spatiotemporal prior helps. Select keyframes deliberately.
Expecting GenSfM to invent a full scene. The priors are object-level and the benchmarks are object-centric (Toys4k, GSO) with four views. Extrapolating to room-scale scenes with twenty views is an untested extrapolation, not a given.
Forgetting that containment is not correctness. Region-selective Gaussian refinement keeps updates inside the target primitives. It doesn't guarantee the new geometry is right when the edit changes topology. Check the unusual angles yourself.
What this means
If you're building a scanning product that needs real-world units, drop the phone-anchor dependency and use the camera IMU with per-segment alignment. A 0.74% error against LiDAR beats 1.94%, and removing a second device from the capture flow removes a whole class of synchronization bugs.
If you're reconstructing objects from a handful of uncalibrated photos, the joint-denoising approach is the one to watch. The 2.8 to 3.8 times Chamfer reduction at four views is a real gap, and the fact that it recovers poses at over 96% Acc@5° without an external pose backbone means you can skip a stage most pipelines still pay for.
If you're editing existing Gaussian scenes, budget the compute. Training-free doesn't mean free, and every trajectory clip is a video diffusion forward pass. Start with structural edits to large, well-observed objects viewed from common angles. Save the topological edits and the odd angles for when you've measured how far the consistency holds.