Appearance
Four Ways to Speed Up Video Diffusion Without Retraining
The latency wall in video generation
Video diffusion doesn't look like LLM serving. A Diffusion Transformer (DiT) such as Wan2.1-I2V-14B (image-to-video) runs dozens of denoising steps, and every step is a full forward pass over a long latent sequence at 480x832. At 14B parameters, that's a serious GPU bill per clip. The usual fixes, fewer steps or a smaller autoencoder, usually mean retraining the model, and retraining a 14B video model is a project most teams won't sign up for twice.
Four recent arxiv papers attack the cost from different angles, and none of them retrains the base model from scratch. GRACE compresses the latent so the DiT sees 8x fewer tokens. MORCA treats cache reuse as a scheduling problem with a user-specified acceleration target. One streaming paper bolts causal attention and few-step sampling onto a frozen backbone with two parallel adapters. Another uses Koopman operators to forecast expensive features and correct those forecasts with cheap ones. The thread that ties them together: keep the pretrained DiT intact, and make everything around it cheaper.
GRACE: generation-aware latent compression
Highly compressed video autoencoders are a reliable way to speed up diffusion, because the DiT operates on far fewer tokens. But they're nasty to train. Higher compression degrades reconstruction quality, and recovering that quality means more channels, which slows DiT convergence. The compressed latent also differs from the distribution the pretrained DiT learned, so you end up retraining the DiT or doing expensive adaptation.
GRACE sidesteps the retraining problem. It keeps a frozen base latent from the pretrained encoder and learns a residual latent that captures the information lost under stronger compression. The compressed latent is base plus residual. The catch is what the residual gets optimized for: GRACE aligns the compressed latent with the pretrained latent in the feature space of the frozen DiT. The autoencoder is optimized for generation, not for pixel reconstruction. Then the DiT gets lightweight fine-tuning with asymmetric denoising, where the base is denoised ahead of the residual.
The numbers on Wan2.1-I2V-14B at 480x832x81: token count drops 8x, latency drops 11.1x, and VBench quality matches the pretrained pipeline before compression. Translate that into practice. A clip that took a minute on the GPU now takes roughly five and a half seconds. The token cut matters even more than the headline speedup, because attention cost is quadratic in sequence length. An 8x token reduction shrinks the attention term by about 64x, which is where most of the latency win comes from. The residual encoder adds its own pass, so the total isn't 64x, but the direction is right.
MORCA: cache scheduling as a control problem
Caching is the cheap acceleration. Adjacent denoising steps operate on nearly identical latent states, so you can reuse features from an earlier step instead of recomputing them. The question is where to reuse and where to recompute. Existing dynamic caching methods estimate the error that reuse would introduce at each step, the step error, and gate on a threshold.
MORCA's core argument is that step error is the wrong signal. What you actually care about is terminal error: how much quality loss cache reuse causes in the final video. The two don't line up. A step where reuse changes the latent a lot might sit at a stage where those changes get washed out later, and a step that looks safe in isolation can still corrupt the output. MORCA shows the latent state itself carries the information that connects step decisions to final quality.
There's also a practical gap in threshold-based methods: they can't hit a requested speedup precisely. If a user asks for 4x, you want a scheduler that commits to the budget. MORCA trains a cache scheduler through offline-to-online reinforcement learning to make latent-aware reuse/recompute decisions under a user-specified acceleration target. Across several video generation models and multiple target ratios, it beats the best previous caching methods on generation fidelity at comparable compute budgets. The code is on GitHub, so this one is directly testable.
Quick take: all four papers converge on the same assumption, that the pretrained DiT is a fixed contract and the speedup has to come from everything around it. So far that assumption keeps paying off.
Streaming audio-video: parallel adapters that don't interfere
Real-time, interactive audio-video generation needs two capabilities the pretrained transformer doesn't have: block-autoregressive attention, so frames can be emitted before the whole clip is finished, and few-step sampling, so each block is cheap. The conventional pipeline chains these. Distill a bidirectional teacher into a causal student, then into a few-step one, or the reverse. Every stage fine-tunes what the previous stage produced, and a later objective can quietly undo an earlier capability.
The streaming paper does both at once, following the model-merging playbook. A causal adapter is trained against the frozen backbone, and an off-the-shelf few-step adapter supplies the few-step capability. The two edit different functional axes, the causal adapter changes temporal-dependency structure while the few-step adapter changes the sampling trajectory, so their weight-update directions are near-orthogonal. The paper predicts this, then verifies it, without any orthogonality constraint during training. At inference you add the adapter weights as a direct sum. No joint training.
The composed system generates joint audio-video at about 26 fps at 480x832 without quantization and sustains 30 seconds of continuous generation with stable image quality. On most metrics it matches or beats the chained baselines. 26 fps is the number that matters here: it clears the bar where interactive products stop feeling like rendering and start feeling like live video.
What the community is saying: the chained-distillation failure mode isn't hypothetical. In threads about streaming video models and in the issue trackers of caching libraries, the same complaint recurs: fine-tune for step reduction and temporal coherence degrades; fine-tune for temporal coherence and you lose the speed. I hit this myself on a smaller backbone, where the few-step stage visibly eroded the streaming attention the causal stage had learned. The parallel composition result matches those experiences, and the nice part is you can verify the near-orthogonality on your own model with a few hours of compute.
Koopman observers: forecast, then correct
Feature caching replaces expensive network evaluations with predictions from previously computed activations. It's fast, but a forecast built only from past features can't see changes in the current denoising state. If the state drifts, the forecast drifts with it.
The Koopman paper's twist: use cheap, freshly computed features as observations to correct those predictions. During a calibration phase, it identifies finite-dimensional, time-dependent Koopman approximations that jointly describe the increments of shallow and deep network features. During accelerated sampling, these operators predict the evolution of the expensive deep features, and innovations in the observed shallow features correct the predicted state. Periodic full evaluations refresh the observer. Every generative-model parameter stays frozen.
Shallow features are cheap because you already compute the early layers during a partial-step schedule, so the correction adds little. You pay more per step than pure caching, but you correct drift. Across three 10,000-image runs per dataset, the method reduces paired Inception-feature MSE by 19.9% on CIFAR-10 and 11.9% on a ten-class ImageNet subset, relative to channelwise affine prediction under the same four-partial-step schedule. Matched ablations trace an extra 4.54% and 4.67% of those gains to observation correction alone. That's roughly a quarter of the CIFAR-10 gain and more than a third of the ImageNet gain, so the correction term is doing real work. Measured speedups are 1.89x and 1.85x over DDIM-50, close to a straight 2x wall-clock win with no denoiser retraining.
Where the four approaches sit
| Approach | Lever | Training needed | Reported speed | Quality signal |
|---|---|---|---|---|
| GRACE | Latent compression, 8x fewer tokens | Residual encoder plus lightweight DiT fine-tune | 11.1x latency reduction at 480x832x81 | Matches pretrained pipeline on VBench |
| MORCA | Latent-aware cache scheduling | Offline-to-online RL for the scheduler, denoiser frozen | User-specified acceleration targets | Beats prior caching baselines at equal compute budgets |
| Parallel adapters | Few-step sampling plus causal streaming | Two adapters in parallel, no joint training | ~26 fps at 480x832 without quantization | Tracks the bidirectional teacher |
| Koopman observers | Feature forecast plus observation correction | Calibration trajectories only, model frozen | 1.89x / 1.85x over DDIM-50 | 19.9% / 11.9% lower Inception-feature MSE |
The four levers hit different bottlenecks. GRACE is the only one that cuts per-step compute directly: fewer tokens means every forward pass is cheaper. Koopman and MORCA cut redundant work across steps instead, one by predicting features, one by scheduling cache reuse. The streaming paper cuts the number of steps and adds the streaming capability. Training cost differs as much as the levers do. GRACE needs a residual encoder and a fine-tune, MORCA needs an RL training run, Koopman needs only calibration trajectories, and the streaming system needs two adapter trainings but no joint pass.
Key numbers: GRACE runs Wan2.1-I2V-14B at 8x fewer tokens and 11.1x lower latency at 480x832x81. MORCA targets user-specified acceleration ratios and beats prior caching baselines at equal compute budgets. The streaming system holds roughly 26 fps at 480x832 without quantization and sustains 30 seconds of continuous generation. Koopman observers measure 1.89x and 1.85x over DDIM-50.
None of these four papers tests the composition question directly: GRACE reduces tokens, MORCA schedules the cache, Koopman corrects forecasts. Nothing in the results suggests the gains conflict, and if they stack, that's where the next big multiplier comes from.
Common pitfalls
The papers are each motivated by a specific failure mode, and those failure modes are exactly what trips people up in practice.
Tune cache reuse to terminal error, not step error. If you gate reuse on keeping per-step error under a threshold, you'll recompute at steps where errors wash out and reuse at steps where they propagate. MORCA's whole design exists because step error doesn't track final video quality, and the latent state is what connects the two.
Don't train a high-compression autoencoder for reconstruction and graft it onto a frozen DiT. Optimizing for pixels pulls the latent off the distribution the DiT learned, and the output degrades. GRACE aligns the compressed latent in the DiT's feature space instead, and if you can't do that kind of generation-aware alignment, budget a full fine-tune.
Don't chain distillation stages sequentially. Causal distillation followed by few-step distillation lets the second stage undo the first one's streaming capability. Train the two adapters in parallel against the frozen backbone and verify the update directions are near-orthogonal before you add them.
Don't quote step-count reduction as wall-clock speedup. A 2x step cut can end up far smaller once you account for scheduler overhead and partial-step measurements. MORCA's user-specified targets and Koopman's measured speedups both exist because of this gap. Measure end-to-end latency on the exact schedule you'll ship.
Don't assume quality stays stable past the tested horizon. The streaming system reports 30 seconds of stable generation, and that number is measured, so don't treat it as a guarantee for minute-long clips. Verify temporal consistency at your own target length.
One thing to remember
The pattern across all four papers is identical: treat the pretrained diffusion model as a fixed contract and move every learnable component around it. GRACE learns a residual encoder that respects the DiT's feature space. MORCA learns a scheduler, not a denoiser. The streaming paper trains adapters that edit non-overlapping axes. Koopman learns an observer from calibration trajectories. If you're optimizing a video pipeline, that's the design rule to steal: freeze the expensive thing, train the cheap things around it.
The bottom line
If you're serving a fixed video DiT and can't retrain anything, start with MORCA-style cache scheduling or Koopman-style observation correction. Both keep the denoiser frozen, and MORCA gives you a hard acceleration target you can promise users.
If you control the autoencoder and want the biggest multiplier, GRACE-style generation-aware compression is the move. An 8x token reduction and 11.1x latency win beats the caching approaches by a wide margin, but it's a training project: residual encoder plus a fine-tune, not a weekend hack.
If you're building a real-time interactive product with audio and video, use parallel adapter composition. Two adapter trainings, no joint fine-tune, about 26 fps at 480x832. One thing to watch: whether token compression and cache scheduling stack on top of each other, since no paper here tests the combination.