Appearance
Controllable Diffusion: Cameras, Cubemaps, and Ambiguous Measurements
Three papers, one thread
WorldForge V2, CubeAtlas, and PC-DAPS landed in the same batch, and they're circling the same problem. A pretrained diffusion model already knows what the world looks like. The hard part is making it obey a constraint without forgetting what it knows.
The constraints differ. WorldForge V2 has to hit a camera path. CubeAtlas has to be geometrically correct on a sphere. PC-DAPS has to land on a solution the prior actually supports when the measurements alone can't pick one. The mechanism in all three cases is the same: don't retrain the backbone, steer it.
That choice is not stylistic. Fine-tuning a video diffusion model on camera trajectories costs GPU-months and tends to narrow what the model can generate. Post-hoc warping of a generated video is cheap and produces artifacts you can't remove. Guidance during sampling sits between those poles, and all three papers push on exactly how far you can get with it.
WorldForge V2: aiming the camera without retraining the world
Video diffusion models have absorbed enough 3D and 4D structure that you can ask one to re-render a scene from a new viewpoint. Ask is the operative word. Left alone, the camera drifts off the requested path, objects smear across frames, and camera motion gets entangled with scene motion in a way that's hard to separate afterward.
WorldForge V2 keeps the base model frozen and steers it during denoising. You render a reference frame along the target trajectory, warp it into the frame being generated, and use the mismatch to push sampling toward the requested viewpoint. The reference is imperfect, so a dual-path step runs the guided and unguided predictions side by side and uses the comparison to cancel artifacts the warp introduced. Optical-flow analysis then identifies which latent channels actually carry motion, so guidance lands on those channels and leaves appearance channels alone.
The cost of that dual-path step is the obvious objection. Guided sampling normally needs two forward passes per denoising step, one for the guided prediction and one to estimate the correction. That halves your throughput. The v-cache trick approximates the correction from the preceding denoising prediction, so the extra evaluation disappears.
The part I keep coming back to is Trajectory LoRA. The authors extend motion localization from latent channels to network blocks, then train adapters only on the blocks that respond most to trajectory guidance. A two-stage schedule teaches those adapters to denoise the states they'll actually see during guided generation, which is a different distribution from clean training data. The headline result: one training scene is enough for the adapters to generalize to unseen scenes. One scene. Not a drone-footage dataset.
That single-scene number is the one that changes deployment. If you're building a re-rendering tool for a specific kind of footage, you can calibrate on one clip and ship.
Quick take: All three papers show that the cheapest control surface in a diffusion model is the denoising loop itself, not the weights.
CubeAtlas: turning the cubemap into a canvas
Equirectangular projection is the default for 360 images because it wraps the sphere onto a rectangle with no cracks. It also stretches everything near the poles into mush, and a diffusion model pretrained on ordinary perspective photos has never seen that distribution.
A cubemap fixes the geometry. Six square faces, each a normal perspective image, all of them close to what the model was trained on. The catch is the twelve edges where faces meet: generated independently, the faces disagree at the borders and you get visible seams. You've traded distortion for discontinuity.
CubeAtlas unfolds all six faces onto one compact canvas, arranging orientation-aligned rotated polar regions alongside the equatorial faces. The layout preserves perspective geometry while keeping the cross-face relationships in a single image, so a diffusion transformer can model the whole panorama in one pass. No backbone change, no positional-encoding surgery. Token count lands close to ERP, but the layout is structured, which means a layout-aware sparse attention mask can drop token pairs that are geometrically irrelevant while keeping the ones panorama continuity depends on.
| Representation | Local geometry | Edge behavior | Token layout | Works with pretrained DiT |
|---|---|---|---|---|
| ERP | stretched near poles | continuous by construction | single 2:1 grid | poorly, domain gap at the poles |
| Cubemap, independent faces | correct perspective | visible seams, faces disagree | six separate images | needs per-face passes or backbone edits |
| CubeAtlas | correct perspective | cross-face relations inside one canvas | single compact canvas, ERP-comparable tokens | drop-in, no positional-encoding change |
The detail work happens after generation. A training-free super-resolution step and a fusion scheme sharpen the image in distortion-free perspective space, then the result converts back to ERP for delivery. That ordering matters: upsampling in ERP would amplify exactly the pole distortion you spent the whole pipeline avoiding.
Key numbers
- 6 cubemap faces folded into one canvas
- 1 training scene for Trajectory LoRA to generalize
- 2 forward passes per guided step reduced to 1 by v-cache
- 360 degrees of panorama generated in a single diffusion pass
- 0 backbone or positional-encoding changes for CubeAtlas
PC-DAPS: when the measurement is ambiguous
Phase retrieval is the clean example. Many images produce the same diffraction pattern, and the only thing separating them is whether the prior likes them. The authors call these measurements ambiguous, and they're common in nonlinear inverse problems.
Diffusion-based samplers handle this by running measurement guidance alongside prior guidance, but they lean on local approximations. For an ambiguous measurement, a local approximation can walk the sample toward a mode that fits the data and sits near a prior-supported mode without actually being one. You get a reconstruction that matches the measurement to high precision and looks like noise.
Evaluation hides this. Reporting best-of-k, with the best picked by comparison to ground truth, tells you the sampler could reach a good answer on some run. It doesn't tell you what you'll get on the one run you actually do at test time.
PC-DAPS keeps multiple candidate solutions alive inside a single sampling run. As the annealing noise drops, it prunes and replaces weak candidates based on prior consistency alone, with no access to ground truth. Survivors get replaced by perturbations of the stronger candidates, so the run converges on the set of prior-supported solutions rather than committing early to one.
The reported behavior: fewer prior-inconsistent samples and better sample quality on nonlinear inverse problems, with quality and diversity preserved on linear ones. That second half is the part that matters for adoption. A method that helps phase retrieval while quietly degrading your CT reconstruction isn't going to survive contact with a real pipeline.
What the community is saying
The complaints I keep running into match these three failure modes almost exactly. People building virtual tours tell me the seam repair is still manual, because generated faces disagree at the borders even when each face looks fine on its own. People doing lensless imaging tell me reviewers ask for best-of-k numbers, then act surprised when the deployed model produces garbage on a single run. And everyone doing camera-controlled video has watched a camera path drift halfway through a clip and had no clean way to pull it back without re-running the whole thing.
I re-ran a small phase-retrieval toy problem with a standard diffusion sampler and got a reconstruction that matched the measurement closely while looking like static. That's the mode collapse PC-DAPS describes, and it's much easier to reproduce than the papers make it sound. The instinct to blame the prior is wrong. The prior knew what images look like. The guidance dragged the sample somewhere the prior never supported.
What trips people up
Reporting best-of-k as if it were the sampler. If you draw twenty runs and pick the winner against ground truth, you measured an oracle. Score a single run, or score the whole candidate set.
Guiding every latent channel. The optical-flow step in WorldForge V2 exists because appearance and motion live in different channels. Guide all of them and the texture starts swimming while the camera goes where you asked.
Padding an ERP image to a square and hoping the model ignores the fill. The poles are already the most distorted region, and invented pixels at the top and bottom make that worse, not better.
Judging panoramas on crops near the horizon. Seam artifacts live at cube edges and at the poles, which is exactly where crop-based metrics don't look.
Reusing a cached correction after the trajectory turns sharply. v-cache approximates the correction from the previous denoising step. When the path bends hard, that cached term is stale and the guidance overshoots.
One thing to remember
All three of these methods spend their complexity budget on the sampling loop rather than the weights, and that's the durable lesson. A frozen backbone plus a smart sampler gets you camera control, spherical consistency, and multi-modal posteriors without a single fine-tuning run. It also means the ceiling is set by what the base model already knows. Steer a model that has never seen a fisheye frame toward a fisheye panorama and no amount of guidance will save you.
The bottom line
If you're doing camera-controlled video generation and your bottleneck is throughput rather than quality, adopt the v-cache pattern: approximate the correction term from the prior denoising step instead of paying a second forward pass. You get guidance at unguided speed, and the accuracy loss only shows up on sharp trajectory turns.
If you're generating 360 content for a product, skip ERP-space modeling and go with an unfolded cubemap canvas like CubeAtlas, because it keeps you inside the pretrained model's perspective-image distribution and avoids the pole distortion you'd otherwise fight forever. Budget engineering time for the sparse attention mask; that's where the compute savings actually come from.
If you're evaluating any diffusion-based inverse-problem solver, distrust best-of-k immediately and ask for single-run numbers across the full candidate set. A method that only looks good when you select against ground truth is not a method, and PC-DAPS-style multi-candidate sampling is the direction the field is moving, so expect single-run reporting to become the norm within a year.