Appearance
The cloud is no longer the default
Three costs push AI back onto your own hardware: latency, privacy, and the per-token bill. A year ago the honest response was "run a small quantized model and accept the gap." That gap is closing, and this week's releases show exactly how.
Five projects landed, one per layer of the local stack. Basalt, a Blackwell-only inference engine, pushes 665 structured tokens per second on a 5090 plus a 5060 Ti. SGLang keeps widening its serving reach across GPUs, TPUs, NPUs, and Apple Silicon. LumaBrowser wraps a whole local assistant, model swapping included, into an open-source app. DigUp runs EmbeddingGemma 2 locally on a Mac and finds any moment in your podcasts and videos. Llama Village runs a 3D world where five llamas with separate memories and goals talk to each other using an on-device model.
Read them together and one architecture keeps showing up. It applies equally to a CUDA kernel and a game loop.
The numbers that matter: Basalt hits 665 structured tok/s on a consumer dual-GPU setup, 2.6x stock Strata on the same weights. DigUp loads a 250 MB text encoder and returns results 0.1 seconds after you stop typing, with indexing peaking under 2 GB. Llama Village keeps five NPCs with separate memories talking via a small on-device model.
Layer one: the engines are getting brutally fast
Basalt started as a Strata fork. Then it rewrote most of the decode kernels and stripped out every GPU architecture except Blackwell. That narrowness is the point. One model (Qwen3.8 Flash-Next), one hardware family, MIT license, weights on Hugging Face, and a .basalt file format that re-packs weights, metadata, vision encoder, and the MTP draft model into a single file per quant.
The headline numbers: 665 tok/s structured output, 354 tok/s prose, 7,317 tok/s prefill at 64k context, at IQ3_XXS and 400 W. Translate that: a 500-token JSON response lands in under a second, prose output outpaces human reading, and a 64k-token prompt, roughly a whole codebase, is consumed in about 9 seconds before generation starts. With eight concurrent users it holds 623 tok/s total, about 78 tok/s per stream, fast enough for chat with no queue.
Single 5090? 585 structured, 316 prose, about 12% slower. The second GPU helps, but the kernel work is doing the heavy lifting.
| Configuration | Structured (tok/s) | Prose (tok/s) | Notes |
|---|---|---|---|
| 5090 + 5060 Ti, IQ3_XXS | 665 | 354 | 7,317 prefill at 64k, 400 W |
| Single 5090, IQ3_XXS | 585 | 316 | about 12% slower |
| 8 concurrent streams | 623 total | n/a | shared KV, MTP on |
| 5090 + 5060 Ti, IQ3_S | n/a | about 300 | about 89% top-1 agreement |
The quant story matters as much as the kernels. IQ3_XXS is the speed lane. IQ3_S keeps about 89% top-1 agreement with the unquantized model at around 300 prose tok/s. Fine-tuned MTP for lower quants is what lets the draft model stay useful when target and draft disagree, and the custom vision encoder runs up to 3x faster than llama.cpp's on GPU, 4x on CPU.
The author's own framing in the release thread: yes, it's vibe-coded, but it's fast vibe-coded, and nobody hand-writes CUDA kernels anymore. Fair. The benchmarks back the attitude.
Quick take: local inference now has two lanes, a fast lane of specialized engines and a wide lane of general frameworks, so pick the lane that matches your hardware, not your hopes.
The serving layer went wide
SGLang is that wide lane. The LMSYS-hosted inference framework serves LLMs, vision-language models, and diffusion models, and its hardware matrix keeps growing: NVIDIA from B200 down to RTX 30/40/50 and Jetson Orin, AMD MI300X and MI350X, Google TPU v6e and v7, Intel Arc and Xeon, Apple Silicon via Metal and MLX, Huawei Ascend NPUs, and Moore Threads GPUs. Integrations in progress add AWS Trainium, Cambricon, Qualcomm, and more.
The same stack that runs in AI labs and cloud platforms runs on a desk. That breadth is the opposite of Basalt's bet, and both bets are correct. Specialized engines extract the last 2.6x on one architecture. General frameworks keep multi-vendor deployments boring, which is a feature.
SGLang also folded diffusion into the engine. Image and video generation share the serving path with text, which matters for agentic workloads that want to emit more than tokens. The RL rollout integrations (verl, AReaL, Tunix) point at another growth area: generating trajectories at scale for post-training.
Layer two: the apps that wrap it all
LumaBrowser is the everything app. A year of development, thousands of installs, now fully open source. The "Anthropic/ChatGPT in a box" claim sounds like marketing until you read the feature list.
Model loading is automatic and tool-aware. Ask for an image and the assistant formulates the prompt, calls the tool, unloads the LLM, loads the image model, generates, then swaps the LLM back. RAM pinning keeps models hot between swaps when you have spare memory, and a custom llama.cpp build keeps the conversation cache warm. A custom routing model decides which tool a request needs, so the orchestration is local too.
It also emits a LAN web server, clusters multiple machines, remote-mounts GPUs from a server instance, imports existing LM Studio installs, and bundles agents, scheduled tasks, a CLI coding agent, and JetBrains and VS Code addons. Too many features, frankly. That's also the point: it's a map of what a local AI OS looks like when the cost constraint disappears.
Single-purpose apps show the opposite end. DigUp runs Google DeepMind's EmbeddingGemma 2 locally to search your files by meaning. Text, images, audio, and video land in one embedding space, so "zebra in a video" opens the clip at the exact moment, "where they talk about sleep" jumps to that minute of a podcast, and "the clause about pets in the lease" opens the PDF page with your words marked. The same search works in Bengali or Arabic. Turn on code search and code: retry with backoff opens the function in your editor.
Three implementation details stand out. The model is a Q8_0 GGUF from ggml-org, 865 MB, downloaded once; search only loads the 250 MB text encoder. Audio and video of any length get chunked into 30-second windows plus a frame per shot, so a three-hour podcast becomes roughly 360 searchable segments. Indexing peaks under 2 GB and the helper process exits when done. All of it sits inside a native Swift app, no Python, with llama.cpp on Metal.
Layer three: a village of talking llamas
Llama Village is where local AI stops being a tool and becomes a place. The developer built it with flutter_scene for 3D and llamadart, his Dart/Flutter binding for llama.cpp, for on-device inference. Five llamas live in the village: Pip, Mo, June, Bramble, and Clover. Each has a personality, needs, relationships, goals, and its own store of knowledge. They make plans, walk around, and stop to talk. A missing scarf, a bread rumor, a secret crush, and the upcoming Berry Festival give them reasons to care.
The architecture is simpler than it looks: one small language model, five different contexts, and game code that owns everything else. The model writes dialogue and thoughts. Embeddings compare what was said against facts the listener could have learned. An optional decision model picks casual topics. Movement, needs, knowledge tracking, and endings run in code. After a conversation, the model reports which facts were mentioned and suggests changes to mood and friendship; the game checks every fact claim before accepting it.
Secrets require explicit keyword evidence. Embedding similarity alone is not enough. So a llama cannot hallucinate its way into knowing a secret. Facts transfer only when the words support them, which means misinformation is possible: gossip can leave a neighbor believing something false, and that false belief persists into later scenes and counts toward the ending.
You play Dash. Click a llama and he flies over; the game generates four things he could say while en route. The game chooses available intentions and facts, the model voices them. Encourage, help, pass news, or start gossip. Trust and mood changes are governed by rules; the model just gives the reply its voice. The Berry Festival closes the week, a winner is chosen by rules, and the game writes a storybook: a journal of notable events, summarized and rewritten by the model into fairy-tale pages with pictures captured during play.
Performance is handled the old-fashioned way: overlap work. The scene keeps running while the model writes, and Dash's dialogue options are pre-generated during his flight. Local inference is fast enough for a game loop if you schedule it like one.
What the community is saying
The Llama Village thread had the sharpest back-and-forth. The fact-checking drew the most attention. The way one developer put it stuck with me: since the game validates claims before a llama learns anything, a llama can't hallucinate its way into knowing a secret. Letting the model give things a voice while the rules own the state feels like the right split, and it's the part most agent projects get wrong.
The practical questions were about the edges. I found myself asking the same things: is a 4k context window enough for NPC dialogue, and what happens when it fills? The journal digest sidesteps it by turning summarization into a feature. Dart isolates per character get expensive; with 50+ actors, pressure shifts from inference to garbage collection. On mobile with 6-8 GB of RAM, q4_k_m is the safer quant, and nobody has a clean answer on whether quantized dialogue quality actually degrades versus the cloud.
One commenter said they'd rather watch than play, at least at first. Watching agents with their own context run into each other is weirdly compelling. Then they'd play Dash as a chaos agent and see how far one false rumor travels before the festival. The follow-up questions were exactly the right ones: do the llamas ever correct a false rumor on their own, and what happens when a fact check passes but a mood update fails? That's the level of systems anxiety you want in a comment section.
What trips people up
Local inference projects keep failing in the same places, and these five projects map them well.
Generating synchronously in a hot loop. If the model and the renderer share a GPU, every blocking generation call drops frames. Llama Village pre-writes Dash's dialogue options during his flight. Overlap generation with anything that can run in parallel.
Letting model output write straight to state. Without validation, hallucinated facts become persistent memories. Llama Village checks every fact claim and demands keyword evidence for secrets, not just embedding similarity. If your agents can learn, the rules must verify what they learn.
Picking the wrong quant for the memory budget. On a 24 GB desktop GPU, IQ3_XXS trades quality for speed and the trade is fine. On 6-8 GB mobile, q4_k_m is the safer floor. Test whether the quality drop actually matters for your use case instead of assuming a global answer.
Forgetting that context windows fill up. A 4k window evaporates after a few multi-turn conversations. Plan for sliding windows or a durable digest. Turn the summarization into a feature like Llama Village's storybook and the constraint becomes a selling point.
Shipping an everything-app without a swap strategy. LumaBrowser's RAM pinning and warm conversation cache exist because model loading is the hidden tax on multi-tool local apps. Without it, every tool call pays a multi-second load penalty and the assistant feels broken.
The fix for most of these is the same: verify in code what the model proposes, and schedule the model like any other expensive resource.
One thing to remember
Every project here works because it split the system in the same place: deterministic code owns state, the model owns language. Basalt's kernels run the decode loop, the quantized model fills in tokens. DigUp's Swift app selects and presents, the embedding model retrieves. Llama Village's rules verify every fact, the language model gives it a voice. Copy that split and your local AI will be predictable where it must be and creative where it can be.
The bottom line
If you're serving multi-user workloads on Blackwell GPUs, use a specialized engine like Basalt, because tuned kernels deliver 2.6x the throughput of a general engine on the same weights.
If your deployment has to span hardware vendors, use SGLang, because one serving stack already covers NVIDIA, AMD, TPU, CPU, and Apple Silicon, with diffusion built in.
If you're building interactive or consumer apps on local devices, adopt the split pattern: rules own state and verify claims, the model only writes language. DigUp proves it at 250 MB of resident memory, Llama Village proves it with five autonomous characters. One thing to watch: orchestration tools like LumaBrowser are consolidating model management fast, and within six months the question won't be which local model to run, but which set of models and how they cooperate.