The cloud used to be the only place a serious LLM would run. That's still true for frontier-scale models, but the interesting work has moved to the edge. In the last few weeks: a home server built from a 2014-era CPU and two repurposed mining cards runs Qwen3.8 27B at 30 to 35 tokens per second. An M5 Max MacBook Pro decodes the same model at 144 tok/s through a new engine called Splash. Apple's M5 Ultra posts Geekbench scores that land near an RTX 4080. And a foundation model called Needle ships as an 8 to 29 MB binary that calls tools on microcontrollers.

None of these match a B200 in a rack, and they don't need to. Each tier solves a different cost equation, and together they show how far the inference stack has come. The same fundamentals run through every layer: quantize the weights, compile the graph, specialize the kernels, match the runtime to the hardware. What changes is how hard you can push each one on the kit you actually own.

The Stack, Layer by Layer

Inference speedup never comes from a single trick. It comes from four layers stacking on each other.

Quantization first. Smaller weights fit more memory and move faster. The home server runs Qwen3.8 27B at Q6K with a mixed-precision KV cache, and Needle ships at 2.125 bits per weight. That's how a 121M-parameter model fits in 29 MB and does the arithmetic of a 50M one.

Graph compilation second. PyTorch 2's torch.compile, built from TorchDynamo and TorchInductor, rewrites normal Python bytecode into a compiled FX graph and generates Triton for GPUs or C++ for CPUs. The paper reports a 2.27x geometric mean inference speedup on an A100 across 180+ real-world models. In practice that's roughly half the wall time from one API call, without leaving eager-mode Python.

Kernel specialization third. TensorRT-LLM is the reference here. Custom kernels for attention, GEMMs, and MoE layers beat generic code by a wide margin on NVIDIA hardware.

Decode acceleration last. Multi-token prediction (MTP) and speculative decoding generate several tokens per forward pass. The home server's MTP toggle alone doubled Qwen3.8 27B output. TensorRT-LLM runs the same idea at rack scale.

The layers compound. Skip quantization and the model doesn't fit. Skip graph compilation and you leave a 2x on the table. Ignore the runtime and the kernel work never reaches the silicon.

The Datacenter End: TensorRT-LLM

At the top of the spectrum sits TensorRT-LLM, fully open source since March 2026. NVIDIA's own results set the ceiling: Llama 4 at over 40,000 tokens per second on B200s, a 1,000 tokens-per-second-per-user barrier crossed with Llama 4 Maverick, and day-0 support for open-weights releases like GPT-OSS-120B and EXAONE 4.0.

The blog archive is effectively a roadmap that smaller stacks follow a few quarters later. Speculative decoding that triples Llama 3.3 70B throughput. KV cache reuse. Disaggregated prefill and decode. Expert parallelism built for NVLink. An MTP implementation for DeepSeek-R1 that mirrors what llama.cpp users flip on with a single flag.

40,000 tok/s is a rack-level number. Divide by concurrency and each user still gets hundreds of tokens per second, which is what serving thousands of agent sessions actually requires. The B200 is the wrong tool for a home lab, but the work it forces, quantized GEMMs, tuned attention kernels, CUDA graph batch sizing, is exactly the work that shows up later in llama.cpp and mobile engines.

One detail to know before you deploy: TensorRT-LLM collects anonymous telemetry by default. No prompts, no weights, but it does report GPU SKUs, parallelism configs, and model architecture classes. Set TRTLLM_NO_USAGE_STATS=1 or DO_NOT_TRACK=1 to turn it off. The tiny-model world has the same default, which I'll get to.

The Garage End: $250 of Mining Cards

The most instructive build in this batch comes from a Reddit user who turned an old PC into a local inference server. Specs: i7-4790K at 4.6 GHz, 32 GB of DDR3, an MSI Z97 Gaming 7 motherboard. The original GTX 1070 died during a repaste. Stripped cooler screws, a slipped screwdriver, and several SMD components knocked off the board. Classic.

I started with CPU-only inference as a fallback. MoE models managed 10 to 20 tok/s, barely usable. Dense models couldn't clear 3 tok/s, meaning a 27B model takes a second or two per word. That's too slow to chat with and miserable for agent loops. So I went hunting on the secondary market and ended up with CMP 50HX mining cards. Nvidia built these for crypto rigs: stripped video outputs, crippled PCIe, deliberately limited compute. $250 bought two of them with 30 GB of VRAM total, a 20 GB modified card plus a stock 10 GB one.

Stock, they were nearly useless. Community driver patches changed the picture. They removed most of the compute limits and added PCIe 2.0 support to cards that shipped with PCIe 1.1. Getting the patches working cost most of a weekend. The first one bricked my driver setup, the second had bugs on the 20 GB variant, and the working guide lived in a GitHub issue buried in the driver repo. It was worth it. With patched drivers and MTP enabled, Qwen3.8 27B runs at 30 to 35 tok/s with prompt processing around 300 to 400 tok/s. MoE models do better: the Ornith 1.5 35B A3B build sits at 80 to 100 tok/s.

$250: what two CMP 50HX cards with 30 GB of VRAM cost in total. 30-35 tok/s: Qwen3.8 27B after driver patches and MTP, double the stock rate. 80-100 tok/s: MoE models on the same cards. 256k context: the max preset, with K and V cache at Q8_0, at 10-14 tok/s. 2 minutes: model switch time from a SATA SSD, solved by a cheap used NVMe.

Quick Take: the gap between unusable and comfortable local inference is rarely the model. It's drivers, quantization, and a decode-side trick like MTP.

The wins came with tradeoffs, and those tradeoffs are the lesson. The 20 GB and 10 GB cards can't split dense models evenly, so tensor-parallel mode hit the smaller card as a hard bottleneck and KV cache space vanished. Patched drivers add PCIe 2.0, but wake-from-sleep crashes them on Z97, so sleep is disabled. Model loading from a SATA SSD took 1 to 2 minutes per switch. Linux cached model files in page cache until the box had 200 MB of RAM left, and GitLab, the web UI, and the desktop session all started lagging. The no-mmap option didn't help. Clearing the cache manually broke llama-server's model unloading, so I had ChatGPT write me a launcher that proxies requests to llama-server, manages per-model presets from .sh files, and clears page cache after each load. It works. It also shouldn't have been necessary.

Apple Silicon Enters the Mix

Geekbench 7 results for the M5 Ultra and M6 surfaced ahead of the Mac Studio and Mac mini launch, and they're strong. The M5 Ultra hit a Metal score of 366,744, about 59% above the M3 Ultra's 230,420 average, and its OpenCL score lands roughly in line with an RTX 4080. The M6, a 2nm chip headed for the Mac mini first, scored 93,217 on Metal, 34% over the M5.

Chip	Metal score
M5 Ultra	366,744
M3 Ultra	230,420
M5 Max	225,606
M2 Ultra	201,144
M4 Max	189,301
M5 Pro	130,896
M6	93,217
M5	69,563
M4	51,818
M1	27,277

Metal scores measure rasterization more than transformer math, so treat them as a rough proxy. The practical data point came the same week: Inco Splash, an inference engine for Apple silicon, runs Qwen3.8 27B at 144 tok/s on an M5 Max MacBook Pro. The developer's claims put it at up to 3x Ollama's decode speed, 2x MLX, and nearly 4x when an agent fans out into sub-agents. 144 tok/s changes the character of local agent work. Point Claude Code, OpenCode, Codex, or Hermes at a local endpoint, and it stops feeling like a fallback. The catch is the floor: M3 or newer, macOS 26.4+, and 36 GB of RAM. This is a big-machine play, not a laptop-for-everyone play.

The Tiny End: 8 MB Models That Call Tools

At the far end sits Needle, a foundation model built for mobiles, wearables, robots, smart home, automotive, and microcontrollers. The whole model is one 8 to 29 MB binary. It won't hold a conversation about poetry, and the team says so openly: they traded general chat capacity to beat models 10x its size on mobile tool calls and match models 2 to 3x larger on extraction.

Tool calling is the design center. Given a set of functions, Needle picks the right one and fills every argument from what the user said. Ask for two things and you get two calls in order. Ask for something no tool covers and you get an empty list, not a hallucinated guess. Every response carries a calibrated confidence score from a learned head, and a byte-level grammar compiled from your schemas guarantees the output parses. Grammar-constrained decoding plus calibrated refusal is what makes tiny models safe to wire into real systems.

The architecture is a Laddered Simple Attention Network. A Monarch Hadamard MLP replaces the feed-forward network, GQA attention gets causal conv taps, and multi-lane hyper-connections mean every depth from 2 to 20 layers deploys as its own model. Most parameters sit in an n-gram engram memory, which is why a 121M model computes like a 50M one. Fine-tuning on one product's tools lifts every subnetwork by 18 to 36 points on DroidCall, and from 4 layers up a tuned subnetwork passes DeepSeek V4 Flash, starting at 29M parameters.

29M parameters is microcontroller territory: kilobytes of RAM, no GPU, no cloud round-trip. Each deployment target ships a prebuilt engine under 1 MB that maps the .cact weight file in place, and engines exist for macOS, Linux ARM, browser/WASI, and bare C. The TinyML research thread makes the same argument. The survey "Tiny Machine Learning and On-Device Inference: A Survey of Applications, Challenges, and Future Directions," which follows the PRISMA review protocol across experimental studies, concludes that on-device inference has a strong record of real-world usability, especially where bandwidth is constrained and responses must be near-instant. For a tool call that returns one JSON object, a local engine beats a network hop every time.

The Full Spectrum

Put the tiers side by side and the pattern is clear.

Tier	Hardware	Representative speed	Rough cost	Sweet spot
Datacenter	B200 + TensorRT-LLM	40,000+ tok/s (Llama 4)	Rack-scale	Serving many users, agent farms
Laptop	M5 Max + Splash	144 tok/s (Qwen3.8 27B)	High (Mac Studio pricing)	Private local agents
Home server	2x CMP 50HX	30-35 tok/s dense, 80-100 MoE	$250 used	Overnight batch jobs, tinkering
CPU fallback	i7-4790K, no GPU	10-20 tok/s MoE, under 3 dense	$0	Emergency mode
Microcontroller	MCU + Needle engine	Tool calls, not tok/s	Dollars per device	IoT, wearables, offline devices

Two things hold the spectrum together. Tool calling is the workload that justifies local inference. The home server exists to give an agent read access to project files. Needle's entire design is function selection. Splash lists agent frontends first, and TensorRT-LLM now publishes posts on evaluating agentic serving with trace replay. And the same four-layer optimization stack applies at every scale, just with different budgets per layer.

Common Pitfalls

Don't pair mismatched cards in tensor-parallel mode. My 20/10 GB split meant the smaller card set the ceiling and ate the KV cache budget. Two identical cards with a clean 1:1 split would have delivered most of the speedup for free. Check memory balance before you buy.

Don't forget that KV cache, not weights, is what eats VRAM at long context. The 256k context preset needed K and V at Q8_0 and dropped speed to 10-14 tok/s. Quantize the weights and leave the cache at full precision, and you'll cap out at short sessions you didn't plan for.

Don't assume environment flags are per-model. GGML_CUDA_DISABLE_GRAPHS=1 applies to the whole CUDA environment and stays active after the model unloads, silently costing 1 to 2 tok/s on every other preset. llama-server's models.ini can't scope it, so I wrote a launcher that handles per-model config outside the server.

Don't run an inference server on a machine that hosts other services without planning for page cache. Linux cached model files in RAM until the box had 200 MB free, and GitLab, Open WebUI, and the desktop started crawling. Clearing the cache after load fixed memory but broke llama-server's unload logic. Budget the RAM or wrap the server in a proxy.

Don't ship without checking telemetry defaults. TensorRT-LLM and Needle both collect usage data unless you opt out. For regulated environments, set DO_NOT_TRACK=1 (and TRTLLM_NO_USAGE_STATS=1 for TensorRT-LLM) before first run, not after a compliance review finds it.

One Thing to Remember

Every speed number in this article comes from the same four layers applied to different hardware: quantization to fit memory, graph compilation to cut overhead, specialized kernels to use the silicon, and decode tricks to get more tokens per pass. The tier you live in decides how much of each you need. The hierarchy never changes.

The Bottom Line

If you're running a local agent over private code or documents, Apple silicon with an engine like Splash is the fastest path. 144 tok/s on an M5 Max makes multi-step agent loops feel synchronous, and the one-command install skips the build-from-source tax that local inference usually charges.

If you're on a few hundred dollars and comfortable with driver archaeology, the mining-card path works. Two patched CMP 50HX cards give 30 to 35 tok/s on a 27B dense model, but plan a weekend for drivers and accept that a mismatched pair will