Ternary Weights Shrunk a 27B Model to 6GB. The Catch Is in the Benchmarks

The wall everyone hits

Here's the problem that defines local LLMs in 2026: the models worth running don't fit on the GPUs people actually own.

Qwen3.8-27B in FP16 needs about 54GB of VRAM. That's an RTX 6000 Ada, a card most people will never buy, or a multi-GPU stack with all the PCIe headaches that come with it. Drop it to Q4_K_M and you're around 16GB, which fits on a 4090 or 3090 but leaves little room for the long contexts these hybrid-attention models are built for.

Ternary quantization changes that math. A 27B model with ternary weights (each weight is one of -1, 0, or +1, packable into two bits) comes in around 6GB at the lowest quant and 7.2GB at PQ2_0, the format used in the community's own testing. Compared to FP16, that's up to 9x smaller. A model that used to require a professional GPU now runs on an RTX 3060 with room to spare, and sometimes inside a browser tab.

This release, the WebGPU demos that followed, and the budget builds it makes worthwhile are all connected. So is the gap between the 98.2% retention headline and what the whitepaper actually shows.

What ternary quantization actually does

Standard quantization (INT8, INT4) keeps the weight distribution but rounds values to fewer bits. Ternary goes further: it collapses every weight to one of three values, -1, 0, or +1. That's one step above binary {-1, +1}, which can't zero out connections at all.

The practical effect is that matrix multiplication with ternary weights becomes mostly additions and subtractions. No multiplications for the weights themselves. On memory-bound hardware, which is most consumer GPUs, that's where the gains come from: you're moving 2 bits per weight instead of 16.

What makes Bonsai 2 notable is that the architecture of Qwen3.8-27B is unchanged. Prism-ML took an existing hybrid-attention causal model and ternarized the weights instead of training a ternary model from scratch. The full attention mechanisms survive, which matters for the WebGPU story, because KV cache behavior dictates how much context you can actually run.

A 27B model at 6-8GB means the weights are no longer the memory bottleneck. The KV cache is. At a 262,144-token context that's a serious chunk of VRAM, and it's now the thing you budget for. The question flips from "can I fit this model?" to "can I fit this context?" That's new for consumer hardware.

The 98.2% headline vs the whitepaper's own numbers

The model card claims Ternary Bonsai 2 retains 98.2% of the intelligence of the full model at 9x smaller. On its face, that sounds like quantization stopped costing anything. It hasn't. The whitepaper that ships with the release tells a different story.

On Terminal-Bench 2.1 and SWE-bench Verified, the two agentic coding benchmarks the release specifically calls out, Bonsai 2 scores 52.8 and 60.8, against 69.7 and 80.6 for the full Qwen3.8-27B. That's not 98.2% retention. It's roughly three quarters of full-precision performance, in the whitepaper's own words.

Benchmark	Qwen3.8-27B (full precision)	Ternary Bonsai 2	Retention
Terminal-Bench 2.1	69.7	52.8	75.8%
SWE-bench Verified	80.6	60.8	75.4%

The gap between the headline and the table is the pattern that matters. An aggregate "98.2% intelligent" number averages over many tasks, and most of those tasks are easy enough that ternary loss is near zero. The hard, agentic, multi-step tasks are where the loss concentrates. Losing a quarter of your agentic coding score is the difference between a model that resolves most issues autonomously and one that needs supervision on a meaningful fraction of them.

The community spotted this within hours of release. The whitepaper also labels the baseline "Qwen3.5-27B," which the source model is not; it's derived from Qwen3.8-27B. People flagged that as a typo. But when marketing math and technical math disagree by 23 percentage points, the technical document is the one to trust.

Quick Take: The memory savings from ternary weights are real and transformative for local inference. The performance retention is closer to 75% on hard agentic tasks, not 98%.

What a real run looks like

I ran Bonsai 2 against two models in the same memory class, all on a single RTX 5090 with a 262,144-token context window, temperature 1.0, and thinking enabled. The prompt was a ~770-word request to build a voxel Japanese pagoda in three.js as a single HTML file. One run per model. No retries, no cherry-picking.

Model	Quant	Size	Output tokens	Time	Speed
Ternary Bonsai 2 27B	PQ2_0	7.2 GB	106,396	~17.5 min	~101 tok/s
Qwen 3.5 9B	Q6_K	7.5 GB	12,105	~70 s	~168 tok/s
Gemma 4 12B	Q8_0	12.7 GB	9,159	~95 s	~96 tok/s

The output quality gap is the headline here. The Bonsai render was substantially more detailed and coherent. The Gemma output looked washed out. The Qwen output was broken. At 7.2GB, Bonsai is producing results that clearly outclass the 9B and 12B models in the same memory envelope. That's the promise of ternary quantization, delivered.

Notice the token counts, though. Bonsai generated 106,396 tokens. The other two produced around 10,000 each. That's not verbosity. It's the reasoning budget: Bonsai ran at xhigh effort (its default) and spent roughly 90% of the run thinking.

The practical upshot: at ~101 tok/s, that 106K-token run took about 17.5 minutes. For a 27B-class model, 101 tok/s is strong. But wall-clock time is the constraint you hit first now, not VRAM. On the other end of the spectrum, Bonsai 1.7B solves simple physics problems at 9.1 tokens per second on a 12-watt Intel N97, the kind of CPU that ships in $150 mini PCs. Ternary scales down even better than it scales up.

The overthinking tax

The token hunger of Bonsai 2 isn't an accident of this release. It's a property of the Qwen3.8 generation, and it's the problem UkisAI built Swift to solve. Swift Qwen3.8 27B penalizes pathological overthinking patterns in small models. Their published results: token usage down 58.3%, speed up 1.95x, accuracy flat. The important detail is that they didn't train the model to think shorter. They trained it to think more efficiently, which is a different and better objective.

The community response was immediate. Downloads jumped from 100k to 150k overnight. Swift became the #1 finetune and #9 overall model on Hugging Face trending. I've generated around 25 million tokens with it since release, mostly on hobby projects. The prompt adherence is what I notice most: I can start a task and walk away, and it does exactly what was asked. It went off the rails once, on a request too high level to hold in the context window. Context compaction at Q4 is weak.

UkisAI is now testing a "Swift Bonsai 2," because their internal testing shows Bonsai suffers from the same overthinking loops. The combination writes itself: ternary weights to fit in memory, efficient thinking to make it fast. If the token reduction carries over, effective throughput roughly doubles, and a ~200 tok/s 27B at 7GB becomes the best local inference deal available.

WebGPU: the browser as the inference runtime

A 6GB model crosses a threshold that 16GB Q4 models never could: it runs entirely in a browser tab. The WebGPU demo for Ternary Bonsai 2 loads the full 27B model client-side with custom kernels. No server, no install, no GPU purchase. MiniCPM5-2B has a similar WebGPU Space on Hugging Face.

Why this works: WebGPU gives browsers direct access to the GPU, and ternary kernels are memory-bound, which maps cleanly to WebGPU's execution model. Hybrid attention keeps the KV cache manageable at long context, so the weight savings don't leak back into context overflow. And when inference happens on-device, the privacy story writes itself. Deployment becomes a URL.

There's friction. The Bonsai GGUF doesn't load in stock llama.cpp; it needs the PrismML fork. The WebGPU demo needs its own kernel implementations. That's the cost of being early. The direction is set: weights below the browser memory ceiling, and the whole local inference stack runs where the user already is.

The other path: 768GB for the price of one GPU

Ternary and WebGPU aren't the only ways the VRAM equation is being attacked. On the other end, I put together a rig from 12x 64GB CMP 170HX cards, the mining cards that lost their job when Ethereum moved to proof-of-stake. 768GB of VRAM total, for less than the street price of a single RTX 6000 Ada.

Key Numbers

• 768GB total VRAM across 12 cards
• Under the cost of one RTX 6000 Ada
• Runs GLM5.3, DeepSeek-V4.1-Flash, Qwen3.8-Flash, Qwen3.8-2.4T, KimiK3, MiniMaxM3
• Inference via vLLM or llama.cpp

768GB runs Qwen3.8-2.4T, the 2.4-trillion-parameter MoE, at low quant. Or a whole fleet of 27B models at the same time. The trade-offs are real: around 5 tokens per second, a lot of noise, a serious power bill, and a room that gets warm. The usual criticisms apply, and I've heard them all: API usage is cheaper, the break-even point is light years away, the whole thing is loud.

The criticism misses what this rig is for: capacity on demand, and the freedom to run the next big model the day it drops instead of waiting for the hardware market to catch up. The used market keeps opening these windows, and the demand for compute isn't shrinking.

Common Pitfalls

Trusting aggregate retention numbers. The 98.2% figure hides a 75% result on agentic coding. Pick a ternary model based on the benchmarks for your actual workload. If you do long-horizon tool use, the whitepaper numbers are the ones that matter.

Forgetting the token tax. A fast ternary model generating 100K+ tokens per task is still a 17-minute wait. Measure time-to-answer, not tok/s. For interactive chat, a thinking-budget-aware sampler or a Swift-style finetune is close to mandatory.

Running stock llama.cpp. The Bonsai GGUF simply doesn't load in the upstream build. You need the PrismML fork or a compatible runtime. Check the model card for kernel requirements before you download 7GB and stare at a cryptic load error.

Assuming weights are the whole memory budget. At 262K context, the KV cache is the new bottleneck. A 7.2GB model can still need 12GB+ total at long context once the full-attention layers do their part. Hybrid attention helps, but budget for it.

Buying mining cards without the full cost. CMP 170HX cards have no display output, need cooling mods, and come from a mining market with unknown card histories. The deal is real, but add the shroud, fans, and PSU headroom to your price before celebrating.

One Thing to Remember

Memory stopped being the binding constraint for 27B-class local inference. The models that win this year pair tiny weights with disciplined thinking budgets, and that merge is already in motion: UkisAI is testing Swift Bonsai 2, and the community is watching for it.

The Bottom Line

• If you're on a single consumer GPU and want 27B-class capability with real context, adopt a ternary model like Bonsai 2; it's the only format that fits, and the ~75% retention on agentic coding is a known, bounded cost you can plan around.
• If you're shipping a privacy-sensitive or client-side product, use WebGPU ternary inference; a model under 6GB runs in-browser on mid-range hardware, and you should budget the custom-kernel porting work until stock runtimes catch up.
• If you're running agentic or batch workloads, measure tokens per task, not tokens per second; a Swift-style thinking-efficiency layer on top of ternary weights is coming, and it should roughly double effective throughput within the next couple of releases.