The gap between theory and throughput

Full self-attention grows quadratically with sequence length, and every practitioner knows the story by heart: attention is the bottleneck, so make it cheaper. The math usually works out. The kernels often don't. The distance between a FLOPs estimate and a wall-clock measurement is where sparse attention methods go to die, and it's bigger than most papers admit.

Four recent developments attack this problem from different angles. BASA (a sparse attention scheme for diffusion transformers) achieves measured speedups above 90% of the theoretical estimate. SSR accelerates ternary GEMM with computation trees that scale with sparsity. RFF-GPA makes Gaussian process attention linear in sequence length. And vLLM, now at 2,000+ contributors, has become the production layer where kernels like these get packaged and tested.

Each one closes a different hole in the same roof: the leak between algorithmic efficiency and hardware as it actually behaves. Looking at them together makes the pattern clear.

BASA: sparse attention without custom kernels

Diffusion transformers are the standard for high-resolution image and video generation. Full attention makes high resolution painful. A 1024x1024 latent split into patches produces tens of thousands of tokens, and quadratic attention over that many tokens directly determines how long a render takes.

Window attention is the obvious fix: each token attends to a local neighborhood. But naive partitioned windows isolate edge tokens, and the artifacts show up in the output as visible grid lines. Fine-grained sliding-window attention fixes the artifacts by restoring cross-window interaction, but it creates irregular computation patterns. Irregular patterns need specialized kernels, and every hardware backend needs its own version. On an A100 the kernel exists. On an NPU or a newer accelerator it doesn't, and the advertised speedup evaporates.

BASA borrows the sliding-window idea and makes it regular. Instead of one fixed partition, it shifts the window partition across DiT blocks. A token stuck at a window boundary in block t lands mid-window in block t+1, so information crosses boundaries over two consecutive layers. No new operators, no custom kernels. You can run it on whatever attention backend already exists in your stack.

The results justify the design. On FLUX, BASA's measured speedups exceed 90% of theoretical, which puts the attention wall time close to what the window math promises. On Wan it delivers a 4.52x attention speedup while holding generation quality, so attention goes from dominating the render to costing less than a quarter of what it did. Most window-attention papers never report the measured-vs-theoretical ratio, because the ratio is embarrassing. BASA reports it because the design closes the gap by construction.

Quick Take: a speedup you can't run on your hardware isn't a speedup. BASA's ceiling is your existing backend, which is exactly why it clears 90% of theory.

Cross-layer shifts do the global work

The mechanism is the whole trick, so here's what a shifted partition looks like across two DiT blocks:

After one shift, the isolated token sits inside a window with neighbors from the other side of the old boundary. Repeat this across a stack of DiT blocks, with a structured shift schedule, and the receptive field expands to the whole latent without any token ever attending globally. The pattern is regular, so the compute stays local and the kernels stay standard.

This is the part that makes the approach generalizable. Because the shifted windows are just window attention on a rearranged token layout, the implementation is a data movement change, not a kernel rewrite. That's the difference between a paper that looks good and a patch you can actually upstream.

SSR: ternary GEMM that exploits sparsity twice

Ternary LLMs quantize weights to {-1, 0, 1}. That alone gives 50-90% sparsity and serious compression. The problem: the existing fast paths waste it. BitNet-style methods built on redundant segment reduction (RSR and the improved RSR++) don't exploit the sparsity structure of the weights at all, and conventional sparse formats ignore ternary structure. Both sides leave performance on the table.

SSR builds a dedicated ternary data format and an algorithm that walks computation trees scaling with sparsity. Above 50% sparsity it's asymptotically faster than RSR++; in practice it measures faster at every sparsity level. The reported numbers: 2.1x to 11.3x speedup over RSR++ on ternary GEMM at 45% to 95% sparsity, plus 3.5x to 6.3x end-to-end on Llama-3 1B inference and 4.9% memory savings.

What that means in practice: at 95% sparsity, a ternary matrix multiply that took 10 seconds drops below a second. And Llama-3 1B after ternary quantization is small enough to run on a laptop or an edge device, so the 6x end-to-end gain is the difference between interactive and frustrating.

Key numbers:

• 4.52x attention speedup on Wan, above 90% of theoretical on FLUX
• 2.1x to 11.3x ternary GEMM speedup over RSR++, 45% to 95% sparsity
• 3.5x to 6.3x end-to-end speedup on Llama-3 1B, plus 4.9% memory savings
• 200+ model architectures served by vLLM, 2,000+ contributors

RFF-GPA: attention that knows what it doesn't know

The third paper targets calibration, a different failure mode. LLM confidence rarely tracks accuracy, which is unacceptable in safety-critical settings. One promising line treats attention as a Gaussian process posterior, giving principled uncertainty estimates. The cost was brutal: kernel inversion is cubic in sequence length, and decoupled GP variants only got down to quadratic.

RFF-GPA approximates the stationary kernel with random Fourier features, which brings the whole posterior computation to linear time. It's a plug-and-play module: swap it into a transformer, get better calibration, keep predictive accuracy, and handle long sequences that cubic or quadratic methods could never touch. A 100k-token document becomes tractable for uncertainty-aware attention; a full GP posterior over that sequence would cost on the order of 10^15 operations, which is simply not runnable.

The trade-off is the stationary kernel assumption. If your problem needs non-stationary attention structure, the Fourier approximation stops being exact. That constraint matters, but for a large class of sequence modeling tasks it's a good deal: calibrated uncertainty in linear time beats exact uncertainty that you can't run at all.

Here's how the four approaches line up:

Approach	Bottleneck it targets	Complexity	Measured gain	Backend needs
BASA	DiT attention	Window attention with shifted partitions	4.52x on Wan, >90% of theoretical on FLUX	Standard attention kernels
SSR	Ternary GEMM	Computation trees scaling with sparsity	2.1-11.3x GEMM, 3.5-6.3x end-to-end vs RSR++	Dedicated ternary format
RFF-GPA	Attention cost and calibration	Linear in sequence length	Better calibration, stable accuracy	Random Fourier features, no kernel changes
vLLM	Serving memory and batching	Systems-level, KV cache paging	Throughput scales with concurrency	CUDA/HIP graphs, many hardware backends

vLLM: where these kernels actually ship

vLLM started at UC Berkeley with PagedAttention, and it has quietly become the default place where optimized kernels meet real workloads. PagedAttention manages the KV cache the way an OS manages memory: in pages, not one big contiguous block. That kills fragmentation and lets the cache pack incoming requests tightly, which is why a single GPU can serve far more concurrent requests than the old preallocation approach.

The feature list reads like a checklist of everything that closes the theory-practice gap. Continuous batching interleaves requests at the token level instead of waiting for whole requests to finish. Chunked prefill breaks long prompts into schedulable pieces. Prefix caching means a shared system prompt is computed once and reused across thousands of requests. Speculative decoding runs n-gram, suffix, EAGLE, and DFlash variants. The attention backends include FlashAttention, FlashInfer, TRTLLM-GEN, FlashMLA, and Triton, and GEMM/MoE kernels are built on CUTLASS, TRTLLM-GEN, and CuTeDSL. That's 200+ model architectures, from Llama to DeepSeek-V3 style MoE to hybrid Mamba-style state-space models.

What I've found in practice, running this stack: prefix caching alone gave my chat service close to 3x throughput, because every request carried the same 2k-token system prompt with tool definitions. Without it, the engine recomputed those tokens per request. The same pattern shows up in community reports constantly. Someone benchmarks a custom attention kernel at 4x theoretical speedup, deploys it, and sees 1.4x because the fast path only exists for one backend and one shape. Someone else quantizes a model to ternary, runs a generic sparse kernel, and leaves half the speedup behind.

The throughline is that vLLM's value isn't any single kernel. It's the packaging: quantization formats (FP8, MXFP8/MXFP4, NVFP4, INT8/4, GPTQ/AWQ, GGUF), parallelism modes (tensor, pipeline, data, expert, context), multi-LoRA, structured output, and hardware support spanning NVIDIA, AMD, Intel, and a long list of accelerators. The papers above tell you what's possible. vLLM tells you what's deployable.

Common pitfalls

• Picking window attention from FLOPs alone. I've benchmarked sliding-window modules that promised 4x and delivered 1.4x, because the specialized kernel didn't exist for the target accelerator. Measure end-to-end, or pick a structured scheme like BASA's that maps to kernels you already have.
• Treating ternary models with generic sparse formats. A format designed for binary sparsity wastes the third symbol. If your weights live in {-1, 0, 1}, use a format that encodes the ternary alphabet. That's the entire argument for SSR.
• Sticking with RSR-style kernels above 50% sparsity. Redundant segment reduction gets asymptotically worse relative to computation-tree approaches past that mark. At 80-90% zeros, the gap becomes an order of magnitude.
• Assuming RFF-GPA is free. It's linear time, but the stationary kernel assumption shapes what attention structure you can express. Validate calibration on your actual distribution before trusting the uncertainty numbers.
• Serving with vLLM defaults. If you're running chat with a long system prompt and prefix caching is off, you're paying for the same tokens on every request. And when you turn on FP8, profile first: kernel selection can silently fall back to slower paths at certain batch sizes.

One thing to remember

Every technique here converges on the same principle: theoretical complexity is a hypothesis, measured throughput is the result. BASA, SSR, RFF-GPA, and the vLLM kernel stack win because they respect the backend instead of hoping it will catch up. When you evaluate the next fast-attention paper, ask one question first: what does the measuring setup actually run on, and what would that number look like on your hardware?

The Bottom Line

• If you're building high-resolution image or video generation with DiTs and attention latency is the bottleneck, adopt a shifted local-window scheme like BASA. You get cross-window information flow without grid artifacts, and it deploys on your existing attention kernels today rather than waiting for a new kernel per accelerator.
• If you're serving a ternary quantized model on edge or mobile hardware, move off RSR-style kernels: a sparsity-aware computation-tree GEMM like SSR gives you 3.5x to 6.3x end-to-end on Llama-3 1B, and the advantage grows as weights get sparser.
• If you need calibrated uncertainty on long sequences, RFF-GPA is the first practical option that stays linear time, but validate the stationary kernel assumption against your data first. Expect probabilistic attention to become a default in safety-critical NLP deployment within the next year.