The benchmark vacuum

I've watched every M5 Ultra benchmark video that's dropped since launch. They follow the same script: unbox the Mac Studio, run an 8B chat model, flash a token counter, call it a verdict on Apple silicon. Those videos measure almost nothing. They can't tell you whether a 256GB machine competes with Nvidia for real serving work, and they don't try.

The LocalLLaMA crowd eyeing this hardware is asking for numbers that matter: 70B-class speculative decoding throughput, long-context prompt processing, multi-user concurrency. What we got instead was influencer content. So the community went back to what it does best: reading changelogs, scraping release blogs, and trying to piece together which levers move inference cost.

That turns out to be a rich question right now. Two things landed this month that most people haven't connected: NVIDIA's Model Optimizer hit its most impressive release cadence since open-sourcing, and Liquid AI shipped day-one speculative decoding for vision-language models on Apple silicon. Add the M5 Ultra hardware question and a long-shot research direction in photonic computing, and you have the full map of inference acceleration.

The four levers

Every inference speedup story reduces to moving less data, moving it faster, or producing fewer serial steps. The concrete versions of that:

Lever	Bottleneck it targets	Evidence so far	Maturity
Unified-memory hardware (M5 Ultra)	Weight streaming from DRAM during decode	256GB fits 70B-class models locally; independent benchmarks still missing	Shipping, poorly measured
Quantization + compression (Model Optimizer)	Memory footprint and memory traffic	Up to 2.6x vLLM throughput, 2-3x smaller checkpoints, accuracy recovered via distillation	Production on vLLM, SGLang, TensorRT-LLM
Speculative decoding (DSpark)	Serial decode latency	1.6x-3.1x decode speedup for under 9% parameter overhead	Production via llama.cpp, MLX, SGLang
Photonic acceleration	Energy per multiply and matmul latency	Lab-stage demonstrations; no credible end-to-end deployment numbers yet	Research

The pattern across every serious release this quarter: nobody wins with a single lever. Liquid AI pairs a drafter with day-one llama.cpp and MLX support. NVIDIA's best ModelOpt results stack pruning, distillation, and 4-bit quantization in one pipeline. The levers multiply, and the teams shipping these speedups design for it from the start.

The hardware lever: Apple's memory gamble

The M5 Ultra with 256GB unified memory is the first Mac that makes local 70B-class inference a real question. A 70B model at fp8 is 70GB of weights. 256GB holds that with room for a deep KV cache, which is what long-context work actually consumes. On the Nvidia side, fitting the same model means an H200-class GPU with 141GB of HBM3e, and the system cost follows the GPU.

Nvidia still wins on raw decode throughput per GPU. Apple wins on a different axis: the whole model lives in one memory pool with no PCIe transfers, and the machine does not require a server room. Which of those matters depends entirely on whether you're serving hundreds of users or running one serious model on your desk.

Reading the LocalLLaMA thread, I feel the same impatience as the person who started it. Every video I click has the same structure: someone who learned the basics last week runs an 8B demo and declares the M5 Ultra either amazing or overpriced. The benchmarks the thread keeps demanding are specific: 70B-class speculative decoding throughput, long-context prompt processing, multi-user concurrency. Nobody with serving experience has posted them yet.

The frustrating part is that the software stack is ready for that benchmark. Liquid AI shipped DSpark support for MLX-VLM and llama.cpp on day one. The tooling exists. What's missing is a credible person running it.

Quick Take: On today's hardware, quantization plus speculative decoding delivers more speed per dollar than waiting for the next chip.

The software lever: NVIDIA Model Optimizer

While the hardware debate runs on vibes, the production speedups that matter this quarter came from software. ModelOpt (rebranded from TensorRT Model Optimizer at the end of 2025) is a library that lets you compose quantization, pruning, distillation, sparsity, neural architecture search (NAS), and speculative decoding, then export a checkpoint ready for vLLM, SGLang, or TensorRT-LLM. The rename matters: this is no longer a TensorRT-only tool.

The flow looks like this:

The headline numbers deserve a careful look:

Key numbers. Nemotron 3 Ultra (550B) at NVFP4 with quantization-aware distillation hit up to 5.9x decode-heavy throughput versus a 754B FP4 competitor while matching BF16 accuracy. A pruned, distilled, fp8 Nemotron-3-Nano-30B-A3B reached 2.6x vLLM throughput and 2.6x memory reduction. Qwen3.6-35B-A3B at W4A4 NVFP4 got 1.30x throughput over BF16 with 3.1x smaller checkpoints. Bielik.AI's Minitron 7B came out 33% smaller, 50% faster, with 90% of the original quality retained.

Technique	What it does	Reported result
PTQ to NVFP4 / FP8	Shrinks weights and activations to 4-8 bits	Qwen3.6-35B-A3B: 3.1x smaller checkpoint, 1.30x vLLM throughput vs BF16
QAD (quantization-aware distillation)	Recovers accuracy lost to aggressive quantization	Nemotron 3 Ultra 550B: 5.9x decode-heavy throughput, BF16-level accuracy
Minitron pruning + distillation	Removes weights, trains a smaller model to mimic the original	Bielik 7B: 33% smaller, 50% faster, 90% quality retained
Pruning + two-phase distillation + FP8	Full stacked pipeline	Nemotron-3-Nano-30B: 2.6x throughput, 2.6x memory reduction

Notice what every strong result has in common: the big multipliers (5.9x, 2.6x) come from stacked techniques, and every 4-bit result mentions accuracy recovery. Plain PTQ at W4A4 degrades models visibly; the gains hold only when distillation or QAD follows. Quantization is a pipeline, not a flag.

The deployment story matters too. These checkpoints run on vLLM and SGLang, not just NVIDIA's own runtimes, so the optimization transfers to whatever you already serve. Adobe's published result is the pattern: 60% lower diffusion latency and 40% lower total cost of ownership from ModelOpt plus TensorRT.

The speculation lever: LFM2.5-VL-DSpark

Speculative decoding is the most surprising lever on the list because it doesn't modify the target model at all. A small draft model proposes k tokens, the big model verifies them in parallel, and you keep the accepted prefix. Verifying k tokens costs roughly one forward pass, so when the drafter is right, you advance k tokens for the price of two passes instead of k.

Liquid AI's DSpark for LFM2.5-VL-3B is the cleanest production example I've seen. The drafter taps the target's hidden states at fixed layers and conditions on them to propose a block of candidates. Image patches and text tokens are projected into a shared representation first, so the same algorithm handles both modalities. The drafter is 280M parameters, 8.9% on top of the 3B target. That overhead buys:

Measured speedups vary a lot by platform, and the variation explains where speculation helps most:

Decode runs 2.30x-3.13x faster on an M5 Max with MLX, 1.57x-2.14x on an M3 Ultra with llama.cpp, and 2.04x-2.66x on an H100 with SGLang. End-to-end gains are smaller: 1.56x-2.62x on the M5 Max, 1.30x-1.77x on the M3 Ultra, 1.64x-2.27x on the H100.

That gap is Amdahl's law. Speculation accelerates decode only. When prefill includes a vision encoder plus hundreds of visual tokens, it can dominate wall time, especially on edge devices. The M5 line's per-core neural accelerators narrow that gap, which is exactly why Apple numbers look competitive despite lower raw bandwidth. For a 3B VLM on a laptop, 1.6x end-to-end can still be the difference between a demo and a product.

The long bet: photonic acceleration

The fourth lever is what "universal photonic artificial intelligence acceleration" points at: doing matrix multiplication in the optical domain. Light passes through a mesh of interferometers or modulators, and the multiply-accumulate happens at the speed of light, with energy per multiply that is, in principle, independent of matrix size.

That physics is real. So are the blockers. Precision sits around 4 to 8 effective bits because the digital-to-analog and analog-to-digital converters at the edges dominate error and power. Packaging has to fight temperature drift. And the entire software stack that makes the other levers work, from vLLM to quantization formats to drafting algorithms, assumes an electronic target.

Watch this one. Don't plan around it. If photonic matmul clears the conversion overhead, it changes inference economics the way the GPU changed training. Until then, treat published results as physics demonstrations, not benchmarks.

Common pitfalls

Five mistakes keep showing up across the threads and changelogs:

1. Benchmarking only decode throughput. The influencer special. Tokens per second hides prefill time, time-to-first-token, and multi-user behavior. Report decode, prefill, TTFT, and end-to-end separately, with batch size and context length stated.

2. Shipping W4A4 without accuracy recovery. 4-bit quantization shrinks the model by 3x, but your eval scores fall off a cliff if you skip QAD or distillation. Every ModelOpt result that looks great pairs PTQ with recovery. Measure perplexity and task evals before and after, not just throughput.

3. Assuming speculative decoding helps every workload. If your wall time is prefill-heavy (long prompts, RAG, vision pipelines), a drafter's decode gains get crushed by Amdahl's law. Profile first. Then add the drafter.

4. Counting parameters instead of memory traffic. MoE models like Qwen3.6-35B-A3B activate a fraction of parameters per token, so a 35B MoE can serve cheaper than a dense 8B. And don't default to fp16 on large models: bf16 has the same footprint and a far wider range on modern accelerators, and fp16 attention values overflow silently in some stacks.

5. Stacking techniques without checking interactions. Pruning, then quantization, then distillation can fight each other. The Nano-30B 2.6x result was a tuned pipeline, not a random combination. Change one thing, measure, change the next.

One thing to remember

Inference speed is a portfolio problem. The best results this quarter came from stacking compression and speculation on top of hardware that already exists: prune, distill, quantize to 4-bit, recover accuracy, attach a drafter, deploy to a runtime that speaks the format. Apple silicon finally has the software stack to support all of that on day one, and NVIDIA shipped a toolkit that makes the recipe repeatable. The M5 Ultra question comes down to one missing artifact: a credible, reproducible benchmark.

Bottom line

If you're serving models in production, start with quantization plus accuracy recovery on your current stack. NVFP4 with QAD gave Qwen3.6-35B a 1.30x throughput lift and 3.1x smaller checkpoints with no new hardware; stacking pruning and distillation pushed a Nano-30B to 2.6x throughput with 2.6x less memory.

If you're building on-device or edge products, speculative decoding is the biggest win per engineering hour. A 280M drafter added 8.9% to a 3B VLM's footprint and delivered up to 3.13x faster decode on the same chip, with day-one llama.cpp and MLX support.

If you're choosing between an M5 Ultra and Nvidia GPUs, don't buy on influencer videos. Test it yourself: 70B MoE with a drafter attached, long context, TTFT and decode reported separately, single and batched. The stack supports that benchmark today. One thing to watch: the window for publishing the first credible M5 Ultra benchmark is weeks, not months, and it will set the narrative for the next year.