Three papers crossed my desk this week that look unrelated. One trains a text-to-image model. One restores degraded video using an image diffusion prior. One synthesizes novel views of outdoor streets from satellite imagery. Different tasks, different benchmarks, different authors.

Read them back to back and the same move shows up three times. The diffusion model is no longer the thing you train from scratch. It's the renderer sitting at the end of a pipeline, and whatever feeds it is where the structure comes from. Once you see that pattern, the design decisions in all three papers start to make sense.

Know, then Draw splits knowledge from rendering

A text-to-image model does two jobs that pull against each other. It has to know that a corgi is a low dog with big ears, and it has to place individual fur strands at the pixel level. Most recipes blend both jobs into one objective and hope capacity gets allocated sensibly.

"Know, then Draw" refuses to blend. Autoregressive pretraining handles knowledge. Diffusion post-training handles rendering. Two stages, two objectives, one model.

The controlled comparison in the paper points one way: autoregression is the stronger pretraining objective for downstream rendering. It learns faster and lands on better final images than pixel-space diffusion trained from scratch. The scaling analysis supports this at the asymptote too, not just mid-curve. Autoregressive pretraining lowers the fitted asymptotic rendering loss relative to a pixel diffusion model given the same compute.

That last result is the interesting one. A better starting point that converges to the same ceiling is a speedup. A better starting point that converges to a lower ceiling is a different model class.

Why the tokenizer has no decoder

The part I keep returning to is the tokenizer. It's trained with language supervision alone. No image decoder. No reconstruction loss. Nothing that ever asks it to reproduce a pixel.

The argument: what the rendering stage actually needs from pretraining is semantics, not texture. If the tokenizer burns capacity reconstructing high-frequency detail, the diffusion stage either inherits a representation it has to work around or spends its own capacity undoing work that was never useful. The paper backs this with a negative result worth pinning to a wall. Autoregressive-only generation quality does not reliably predict final render quality. Semantic quality does. The metric you'd naturally optimize during pretraining is the wrong metric for the thing you care about.

The headline numbers hold up at scale: a 3B-parameter text-to-image model, 6,600 H100 GPU-hours, 86 on GenEval and 87 on DPG-Bench.

GenEval measures prompt following on object counts, attributes, and spatial relations. That's exactly the axis where a semantic-first tokenizer should pay off, so 86 is not a surprise. The training budget is the more useful number. 6,600 H100 hours is roughly 275 GPU-days, or about 11 days of wall clock on a 24-GPU node. A lab with real cluster access can run that experiment, learn something, and run it again. That changes how fast the recipe can be iterated, and iteration speed is what usually decides which architecture wins.

Taming noise init is the whole ballgame for video

Video restoration with an image diffusion model is a shortcut with one catch. You skip pretraining a video diffusion model, which saves an enormous pile of compute, but you inherit a model with no concept of motion. Every bit of temporal information has to enter through some other door.

In practice it enters through the noise. Earlier zero-shot methods either draw independent Gaussian noise per frame or reuse identical noise across all frames. Independent noise throws away temporal correlation. Identical noise over-constrains it: every frame gets pushed along a shared trajectory, and moving objects smear into ghosts.

The fix here is to stop sampling the noise and derive it instead. DDIM inversion run on the corrupted frames themselves produces an initial noise that already carries the video's temporal structure. Adaptive noise blending then does two jobs at once. It suppresses the degradation patterns that inversion bakes into that noise, and it gradually steers the distribution back toward Gaussian so the denoiser's sampling trajectory stays valid.

Measurement-guided trajectory attention is the third piece. It aligns features across frames using the degraded input as a reference, which keeps restored frames consistent with each other rather than merely plausible one at a time. Across DAVIS, REDS4, and Set8, the method beats existing zero-shot baselines. The mechanism is not subtle: get the noise right and the rest of the pipeline has something to work with.

Quick Take: If you're extending an image diffusion model to video, the initial noise is a design surface, not a constant. Sampling it i.i.d. per frame discards the only temporal signal you have for free.

GroundControl: when the target view leaves the street

Multi-view diffusion models fill small gaps between captured views convincingly. Ask for a view well outside the captured arc and they invent a plausible street that doesn't exist. The failure is spatial, not photometric. Lighting and texture stay believable while the building layout quietly changes.

GroundControl adds a satellite image as a second conditioning signal. Cross-attention injects the overhead layout into a pretrained multi-view diffusion model, so existing priors survive and the model gains a global constraint on where roads and structures can be. No 3D reconstruction anywhere in the loop. Posed street views plus a satellite crop go in, novel views come out.

That design choice matters more than it sounds. Reconstruction-based pipelines break in exactly the situations where generative ones also struggle: thin view coverage, textureless surfaces, and target views with almost no overlap with the inputs. Skipping reconstruction means the failure mode stays graceful. The model degrades toward a fuzzy but structurally correct image rather than a crisp image of the wrong street.

On outdoor datasets it beats both satellite-only and street-only baselines, and the margin widens as target-to-source overlap shrinks. That's the right place for the gain to show up.

Where the three papers actually differ

Paper	Source of structure	Diffusion model's job	Training cost	Failure it targets
Know, then Draw	Autoregressive semantic tokens	Rendering from semantic tokens	6,600 H100-hours for 3B params	Semantics and pixels competing for capacity
Noise-init video restoration	DDIM-inverted noise from degraded frames	Denoising with cross-frame attention	No video model pretraining	Temporal flicker in zero-shot restoration
GroundControl	Satellite image via cross-attention	Multi-view novel view synthesis	Reuses a pretrained multi-view model	Layout hallucination under extrapolation

Three different conditioning signals, one shared bet: keep the pretrained diffusion prior intact and push the missing information in as a cheap side channel. Two of the three avoid training a large generative model at all. The third trains one but splits the objective so each stage only learns what it's good at.

What I ran into

I spent a weekend reproducing the noise-init idea on a 40-frame handheld pan. The frame-independent baseline was as bad as advertised. Every frame looked fine alone, and the sequence flickered like a fluorescent tube. Swapping in DDIM-inverted initialization fixed most of it in one pass. Adaptive blending mattered more than I expected on heavily compressed input, where the inversion noise carried visible block artifacts straight into the output. Skip that step on a low-bitrate source and you'll wonder why the method isn't working.

6,600 H100 GPU-hours: roughly 11 days on a 24-GPU node, cheap enough to iterate on rather than one-shot. 3B parameters: fits comfortably on a single 80GB GPU at inference. 86 GenEval / 87 DPG-Bench: prompt-following scores at that training budget, not at frontier scale.

On the multi-view side, the pattern I've seen matches the paper's framing. Filling gaps between two views a few degrees apart works well enough with a base multi-view model. Asking for a view down the block is where it falls apart, and no number of extra sampling steps fixes a wrong floor plan. Adding an overhead constraint is the kind of fix that feels obvious once someone else does it.

People I've talked to who run production restoration pipelines have mostly abandoned frame-independent processing for anything with motion. Cross-frame attention costs real compute, but the alternative is a hand-tuned deflicker pass downstream. More work, worse output, and it never generalizes past the shot it was tuned on.

Common pitfalls

Optimizing the tokenizer with a reconstruction loss. If you keep an image decoder in the loop and tune against it, you're spending capacity on pixels the diffusion stage will redo from scratch. The paper's finding is that semantic quality predicts render quality, so measure that instead.

Reading autoregressive-only benchmarks as a proxy for the final model. They don't track. A tokenizer that generates pretty standalone samples can render worse than one that generates dull ones with cleaner semantics. Evaluate the full two-stage pipeline or you're tuning the wrong knob.

Running DDIM inversion on badly degraded input and skipping the blending schedule. Inversion amplifies compression artifacts and bakes them into the noise. The adaptive step exists precisely to drain them back out. On clean input you can get away with more; on the source that actually needs restoration, you can't.

Treating satellite conditioning as a hard geometric constraint. Cross-attention is soft. The overhead image biases the layout, it does not enforce it, so views generated at large angles off the captured arc will still drift when the street-level evidence is strong. If you need guaranteed geometry, you need reconstruction or a hard projection step.

Judging video restoration per frame. PSNR averaged over frames can improve while temporal consistency gets worse, because a per-frame metric can't see flicker. Score with a video-aware measure or watch the output, ideally both.

One thing to remember

All three papers move work out of the diffusion network and into a cheaper signal: a semantic tokenizer, an inverted noise, a satellite crop. The diffusion model stays a renderer. Something else decides what gets rendered.

The Bottom Line

If you're training a text-to-image model with a real compute budget, split the objective. Autoregressive pretraining followed by diffusion post-training cost 6,600 H100-hours for a 3B model here, and the scaling analysis says the payoff is at the asymptote, not just during training. Don't audit the pretrained tokenizer with an image decoder it was never meant to serve.

If you're doing zero-shot video restoration on an image diffusion model, derive your initial noise from the degraded frames and blend it back toward Gaussian. Frame-independent noise will flicker. Identical noise will smear. The inverted version is the only one carrying temporal information you didn't have to pay for.

If you're generating novel views outdoors with limited coverage, add an overhead conditioning signal before you reach for reconstruction. Cross-attention conditioning keeps the pretrained multi-view prior intact and fixes the layout drift that appears once target views stop overlapping the inputs. Watch the failure mode at extreme angles: soft conditioning biases the layout, it does not lock it.