Why local AI suddenly feels fast

A 322M-parameter model just delivered a structured decision in 7.39ms, end to end, in 687.6 MiB of peak memory, on a laptop. It never generates text. That design choice, skipping autoregressive decoding entirely, is what makes the rest of the numbers possible.

For most of the past two years, "runs on my laptop" meant either a small model that drips out a few tokens per second or a bigger one that pins the fans at full speed. The bottleneck was never the GPU alone. It was the whole path: weights that barely fit in memory, a generation loop that makes the CPU wait on the GPU, a tokenizer that starves the model, and autoregressive decoding that spends compute on every token.

A handful of projects attacked different layers of that path in the same window. Laya-MLX brought a non-autoregressive decision model under 1GB. transformers started loading GGUF checkpoints natively, reusing llama.cpp's Metal kernels. tokenizers v1 made encode 3 to 30 times faster. And a Waterloo research project, ProgramAsWeights, compiles English task descriptions into small local programs.

The numbers that matter. Laya-MLX peaks at 943.6 MiB for the 421M English checkpoint and 687.6 MiB for the 322M multilingual one. P50 latency on an M3 Max: 13.42 ms and 7.39 ms, end to end. At batch 50, throughput reaches 395 questions/s. A 0.6B PAW interpreter hits 73.4% exact match on FuzzyBench where direct Qwen3-32B prompting gets 68.7%. tokenizers v1 encodes 3x to 30x faster than v0.23 with identical token IDs.

None of these is a headline model release. The pattern matters more: the serving path, not the parameter count, is where the wins are hiding.

The architecture hack: skip text generation

Jev went viral a week before the MLX port landed. It does one thing: given a state and a question, output a decision. No chat, no code, no prose. Cutting the per-token generation loop removes the most expensive operation in modern LLMs.

Laya is the open-source sibling in the same spirit: a multilingual, non-autoregressive System 1 decision model. It handles three task types: choice (pick one of several options), score (rate by a rubric), and noul (estimate the probability that a statement is true). One forward pass, structured output, no text.

That changes the memory math. A 421M model that never generates text doesn't need a KV cache that grows with output length, and its weights sit comfortably in the unified memory of any Apple Silicon Mac. The real constraint is the context window: 512 tokens for the English checkpoint, 1024 for the multilingual and typed-decisions variants. State, instructions, and options all share that window. This is built for short, repeated, well-defined decisions, not long-form reasoning.

Checkpoint	Parameters	Peak MLX memory	P50 latency (M3 Max)	Context
Laya English	421M	943.6 MiB	13.42 ms	512 tokens
Laya multilingual	322M	687.6 MiB	7.39 ms	1024 tokens
Laya typed-decisions	421M	n/a	n/a	1024 tokens

Memory and latency were reported for the English and multilingual checkpoints; the typed-decisions variant targets structured decision workflows. Latency here is not a kernel benchmark. The measurement includes prompt preparation, tokenization, tensor construction, synchronous inference, calibration, and result formatting; only model loading is excluded. A 7.39ms round trip is faster than a blink, fast enough to gate a keystroke or a UI action.

The original Laya numbers put the advantage in perspective. Third-party Jev latency on that workload was 236 to 276ms. Laya measured 32.8ms for a single question on a Tesla T4, 7 to 8 times faster, and a single T4 processes hundreds of batched questions per second. Remove generation and the remaining inference is cheap enough to run hundreds of times per second, wherever the device happens to be.

The MLX port: 1GB and 7ms

mizorewww reimplemented Laya's full network on Apple MLX, so PyTorch and the Transformers runtime can go away. No cloud API, no dependency chain. The checkpoint downloads once; after that, inference stays on the machine.

Porting to a new stack has a failure mode: it produces the same output by coincidence. Laya-MLX was verified across all three checkpoints in FP32 and FP16 on 63 validation questions. All 378 comparisons matched the original Laya outputs. No drift.

The speed holds up in real use. The repo ships a snake game where every move calls the model for a decision, with a separate cycle safety layer that corrects dangerous actions. With compilation and prefix-reuse optimizations enabled, a 2400-step run on an M3 Max sustained 75.40 moves/s with zero deaths. The safety layer visibly intervened twice. That's a model making roughly 60 decisions per second, under control, inside a game loop.

import laya_mlx as laya

agent = laya.load("aac6fef/laya-mlx")
agent.predict("state here", "Is this email urgent?")

Reading the launch coverage, the running joke was that a sub-1GB local version appearing a week after the original went viral is basically free. I felt that. The port landed so fast that the original model's release notes were still circulating, and more than one commenter noted that BERT probably recognizes the job description, since encoder-only models have been making non-autoregressive decisions for years.

Quick Take: The trend to watch is not any single checkpoint; the entire serving path, from weights to runtime to tokenizer, is getting slim enough to live on one machine and answer in milliseconds.

Compile once, run forever: ProgramAsWeights

ProgramAsWeights (PAW) from the University of Waterloo attacks a different problem: what if the model that understands your task never has to run again after setup?

The idea is to split compilation from inference. A finetuned Qwen3-4B compiler takes an English function description and produces a LoRA adapter for a frozen Qwen3-0.6B interpreter. A "neural program" is that adapter plus a pseudo-program: a cleaned-up task description with a few input/output examples that ships in the interpreter's prompt.

The part the author still finds surprising is how much the frozen interpreter changes when you swap the adapter. I found the same when testing the recommended workflow: write a muddy specification and the compiler produces a muddy adapter; clean up the spec, add a few examples, and the error cases disappear. Hand-writing a small validation set before compiling anything is the fastest debug loop I've used in this stack.

Training routes gradients through the frozen interpreter into the compiler's adapter-generating layers. The interpreter weights never move; the compiler learns how to specialize them. Compilation takes seconds, and after it finishes, the big compiler model is not needed again.

The results make this more than a research curiosity. On FuzzyBench, a synthetic suite covering classification, extraction, parsing, and format conversion, the 0.6B interpreter with a compiled adapter reaches 73.4% exact-match accuracy. Direct prompting of Qwen3-32B scores 68.7%. A 0.6B model beats a 32B model on the same function while running on a CPU.

In fairness, FuzzyBench is synthetic and these are the project's own reported numbers. The comparison will get picked apart. But the structure of the result should survive scrutiny: a small model plus a compiled adapter can beat a big model plus a prompt on a narrow function, at a fraction of the compute.

The follow-up, Compile by Training, pushes further. Teachers synthesize task examples, then the generated adapter is finetuned for about 100 steps, roughly a minute of compilation. On FuzzyBench-Hard, the subset where the original compiler produced no exact matches, that mode reaches 83.6% semantic accuracy. Both compilers produce the same reusable program format, and the teacher models leave the picture at compile time.

The longer-term promise: large models become tool builders. You describe the function, get a small neural program back, and compose it with ordinary code. That is a different distribution model for on-device AI.

GGUF support lands in transformers

Meanwhile, Hugging Face shipped GGUF loading directly into transformers. You pass gguf_file to from_pretrained and the weights stay packed on Metal. No dequantization step, no llama.cpp runtime, just the standard transformers API on Apple Silicon.

The trick is reusing llama.cpp's ggml Metal kernels through a new kernels library. The quantization kernel reads packed quantized weights directly, including the selected experts in an MoE model. A fused normalization kernel handles the zero-centered RMSNorm in newer Qwen architectures. The attention kernel provides Metal flash attention. A gated-delta-net kernel accelerates the linear-attention layers in hybrid models. A custom Metal top-k kernel handles MoE expert routing. Together they cut the GPU work per generated token.

For picking a quantized checkpoint, the sizing math is straightforward:

GGUF variant	File size	Tradeoff
BF16	8.42 GB	Unquantized reference
Q6_K	3.53 GB	More precision than the smaller variants
Q5_K_M	3.14 GB	Middle ground between size and precision
Q4_K_M	2.74 GB	Practical starting point for local inference

For a 4B model, Q4_K_M at 2.74 GB leaves most of a 16GB MacBook's unified memory for the OS and the actual workload. The docs suggest starting there, then trying Q5_K_M or Q6_K if memory allows, and evaluating on the work you actually intend to do rather than a generic benchmark.

Two generation-loop changes in transformers benefit every model, not just GGUF files. An unneeded attention mask is dropped early when the input has no padding, and the stopping check is deferred asynchronously so the CPU keeps scheduling GPU work while the GPU runs. Each removes a small per-token stall.

Benchmarks against llama.cpp are close on decode throughput across a small dense model, a larger dense model, and an MoE checkpoint. The conditions are not identical: the transformers measurement includes prefill, while llama-bench reports decode-only throughput. One of the Hugging Face founders ran a 27B coding agent locally via llama.cpp and said that on non-trivial codebase tasks it felt close to the latest Opus. The comparison is subjective, but the direction is explicit. The role split is deliberate: llama.cpp stays the dedicated inference engine, while transformers gives PyTorch users hooks, evaluation tooling, and custom generation loops on the same GGUF weights, plus a transformers serve endpoint that speaks the OpenAI API.

The boundaries are real: the packed path is MPS-only for now, padded batches don't get the mask shortcut, and architecture coverage starts with the Qwen3.5 family.

The tokenizer was the hidden bottleneck

Tokenizers get the least attention and cause some of the worst stalls. When a model generates 100 tokens per second, the GPU can sit idle waiting for the CPU to tokenize the next prompt. tokenizers v1 directly attacks this.

v1 produces exactly the same token IDs as v0.23. No retraining, no output drift, same API. The improvements are structural:

Change	What it does
bitcannon	Replaces regex splitting with SIMD bitstream operations; 64 bytes per register op instead of a character-by-character scan
word cache	Thread-local memo from pre-token bytes to finished IDs; repeated words skip the merge process
merge-loop rewrite	Intrusive doubly-linked list in a caller-owned scratch buffer; merging updates indices, never allocates
native parallelism	One shared tokenizer encodes from many threads; scratch buffers and caches come from per-thread sub-pools
workspace split	One crate becomes a workspace; tk-encode is the only required runtime dependency

On an Apple M4 Max, single-threaded encode is 3 to 30 times faster than v0.23 across ten model families, with t5-base at the low end and gpt2 at the high end. Scaling holds at 76% of linear across eight workers, so one shared tokenizer can feed multiple generation streams without becoming the queue.

The caveat matters. bitcannon only covers specific byte-level patterns (GPT-2, cl100k, o200k, Tekken, DeepSeek). Tokenizers outside those grammars keep the regex path and get none of that speedup. And benchmark methodology dominates results: repeatedly encoding one cached document looks artificially fast. The tokbench suite uses distinct documents per iteration, and the headline numbers reflect that.

For local inference, the practical result is that the tokenizer stops being the thing that starves the model. Decoding also got faster through reusable buffers and batched parallel decode, which matters for any workload that renders long outputs.

One stack, several owners

The last piece is organizational. Jun Kim, creator and maintainer of oMLX, joined Hugging Face to support the MLX community. oMLX stays Apache 2.0 and Kim keeps leading it. One stated focus: making it quick to go from a transformers model definition to a reference MLX implementation that different engines can consume. Hugging Face already works with the mlx-lm, mlx-vlm, and LM Studio teams, and expects oMLX to be a testbed for ideas that get upstreamed where they make sense.

Read those moves together and a strategy emerges. Hugging Face is pulling the local stack under one roof: GGUF loading in transformers, ggml Metal kernels distributed through a kernels library, a faster tokenizer, and funded MLX tooling. Each release removes a format conversion or a runtime switch that used to eat a morning.

Common pitfalls

A few concrete mistakes showed up across these release notes and threads.

1. Choosing a quant by size alone. Q4_K_M is a starting point because of the file size, not a guarantee. Loss depends on both the model and the task, so benchmark the quantized checkpoint on your actual workload. For a 4B model, the memory saved by dropping from Q6_K to Q4_K_M rarely makes up for the output difference.

2. Assuming "runs under 1GB" means full capabilities. Laya's 512-token context has to hold the state, the instructions, and the options together. It handles short, high-frequency decisions, not document analysis or multi-turn conversation. Longer inputs get truncated, and nothing warns you.

3. Benchmarking Apple Silicon without cooldown. Back-to-back generation runs on an M2 Max decay by 10% or more from thermal throttling; the transformers benchmark script sleeps 90 seconds between runs. If your latency numbers keep shrinking run after run, suspect heat before suspecting a model improvement.

4. Expecting tokenizer speedups everywhere. bitcannon covers specific BPE patterns. Models outside the supported grammars keep the regex path and get none of the advertised gains. Check the pattern before promising anyone a 30x tokenization win.

5. Trusting a port without verification. Laya-MLX passed 378 output checks across FP32 and FP16 because the author built verification into the port. If you move a model to a new backend, do the same with fixed validation questions and both precisions. Output drift between backends is silent.

One thing to remember

Every project in this cluster attacks the same enemy: idle silicon. Decision models skip generation entirely. GGUF loading keeps weights packed so memory bandwidth is not wasted expanding quantized tensors. The tokenizer rewrite keeps CPUs from starving GPUs. PAW compiles the expensive part once so the cheap part can run anywhere. When you evaluate an on-device model, measure the whole path end to end on the machine you will ship, because the bottleneck is never where the README says it is.

The bottom line

1. If you're building a high-frequency decision service, email triage, content moderation, form validation, or game logic, a non-autoregressive model like Laya or Laya-MLX is the right default: under 1GB of memory, sub-10ms decisions on Apple Silicon, hundreds of batched calls per second. Skip the