Running Qwen 3.8 Flash-Next on a 12GB Card: What the Local Stack Looks Like Now

A thread on r/LocalLLaMA asked this week whether Qwen 3.8 Flash-Next at Q2 is better than the 27B dense model at Q4. The thread didn't settle it. But around it, three posts landed that give us the pieces of an answer: a custom inference engine that pushes Flash-Next to 65 tokens per second on a 12GB card, an experiment that grafts Flash-Next's n-gram memory onto a 0.8B model for a 5.05% validation perplexity cut, and a painfully familiar demo of the 27B burning tokens on an internal monologue about running a test suite.

None of these touch the backbone weights. That's the pattern worth noticing.

What Flash-Next is before you squeeze it

Flash-Next isn't a normal dense transformer. It pairs a sparse MoE with a pretrained n-gram memory called PLE, roughly 51B parameters of lookup tables that store sequence statistics from pretraining. Quantized to Q2_0, the whole thing needs 37.6GB across system RAM and VRAM, plus 0.91GB if you want the vision encoder. That's a footprint bigger than the 12GB card it runs on, and the engine still gets 65 output tokens per second out of it.

The PLE being a sidecar matters more than it sounds. Lookup tables quantize differently than attention weights, and because they're frozen, you can treat them as a portable asset. That's the foundation for the Qwengram experiment below.

Key numbers

• 65.1 tok/s: Flash-Next output at Q2_0 on a 12GB RTX 5070, roughly 4x the 15 tok/s the author got from llama.cpp a few weeks earlier
• 543 tok/s: prompt processing at Q2_0, up from 100 to 120 tok/s in llama.cpp. An 8K prompt preloads in about 15 seconds instead of 70
• 37.6GB: combined RAM + VRAM floor for Q2_0. A 12GB card alone leaves you roughly 26GB short
• 5.05%: validation perplexity drop on the 0.8B model after grafting the PLE
• 99.1%: of the reader's NLL gain retained when the PLE sidecar runs at Q8_0 instead of BF16

65 tokens per second on a 12GB card

The Strata engine is a from-scratch CUDA runtime built for this one model and this class of machine. The author's earlier llama.cpp setup did 15 tok/s output at IQ3_XXS. Strata does 44.8 tok/s at the same quant, and the faster quants do better still. The model file comes from ISTA-DASLab, which published RCO-GSQ quants tuned specifically for aggressive compression. The numbers below are from a 128K context run on a 12GB RTX 5070, a Ryzen 5 7600, and 64GB of DDR5-5600.

Quant	Output (tok/s)	Prompt processing (tok/s)	Min RAM + VRAM	Rough unsloth equivalent
Q2_0	65.1	543	37.6 GB	Q3
IQ2_XS	52.0	472	39.2 GB	Q4
IQ3_XXS	44.8	414	47.0 GB	Q5

The vision encoder adds 0.91GB on top. Don't skip the memory column when you read this. Q2_0 needs 37.6GB combined, which means roughly 26GB of system RAM is doing active work while the GPU holds what it can. On the author's setup, 64GB of DDR5 at 5600MT/s is load-bearing. On a 32GB machine or a single-channel config, expect the offload path to become the bottleneck and the speeds to drop.

At 65 tok/s, generation runs at roughly the pace of someone reading aloud. Chat feels instant, and long-form code or prose stops being a waiting game. At 543 tok/s prompt processing, an 8K prompt preloads in about 15 seconds, which changes how you work: you can afford to stuff the context window with retrieval content instead of carefully trimming it.

Quick Take: the through-line across all three posts is that nobody fine-tuned anything. The wins come from orchestration, memory grafting, and gating decisions. Local inference is turning into an assembly problem.

The Q2 vs Q4 question nobody answered

The thread about whether Flash-Next at Q2 beats the 27B at Q4 got engagement but no measurements. Let's do the hardware math the thread skipped. A 27B dense model at Q4 is over 14GB of weights before GGUF overhead. On a 12GB card, that overflows before you account for KV cache, so you're offloading part of the model to system RAM. Offload-bound generation on a 27B usually lands in single-digit or low-teen tokens per second. That's the direction of reported numbers, not a measurement; nobody in the thread posted a full apples-to-apples run.

I ran the 27B in thinking mode around the same time and it produced the exact caricature everyone is joking about. Hundreds of tokens of "I am now going to run the tests, for real this time, no more analysis," a Morgan Freeman voiceover in my head saying it did not in fact get on with it. The underlying issue is real: a reasoning model at single-digit tok/s will happily spend several minutes convincing itself to act.

The capability side of the comparison favors the MoE for two structural reasons. Active parameters per token stay small, on the order of 3B for the MoE class these models belong to, so quantization pressure spreads across a large pool of weights the model rarely needs all at once. And the n-gram tables are lookup statistics rather than learned interactions, so aggressive quantization damages them less. The Q8_0 retention result from the Qwengram experiment supports that second point.

Grafting a 51B-parameter memory onto a 0.8B model

The Qwengram experiment is the most interesting piece of the cluster. The idea: take Flash-Next's pretrained PLE n-gram memory, freeze it, and bolt it onto a Qwen3.5-0.8B backbone that is also frozen. The only trainable parts are two small readers with rank R=1, placed at decoder layers 3 and 9, plus a token-dependent linear gate that decides how much memory to inject at the later layer. Training ran on free Kaggle notebook GPUs for 15M tokens. No backbone fine-tuning at all.

The result: validation NLL dropped from 2.9056 to 2.8538, and perplexity from 18.28 to 17.35. That's a 5.05% reduction. The author is explicit that this is a language-model validation result, not a 5% benchmark accuracy claim.

A few findings shaped the final design, and they double as a recipe for anyone trying to copy this:

• The real pretrained PLE beat both random-memory and permuted-memory controls. The graft carries real structure, not a generic lookup crutch.
• Reader loss kept improving well past 5M tokens. The 20M-token reader improved aggregate LM loss further, but regressed on math. The 15M checkpoint is the balanced one.
• Fixed late-layer memory injection hurt LAMBADA. Token-level arbitration that varies per token recovered most of that tradeoff.
• The gate is not a learned constant. Memory strength varies substantially from token to token, and treating it as fixed is what hurt the earlier ablation.
• With the same memory budget, learned token placement beat shuffled placement. Routing memory toward high-uncertainty tokens helped, but didn't match the learned gate.
• A warm-started R=4 reader produced a small aggregate gain but introduced code and math regressions. R=1 stayed.

That last pair of findings is the part worth dwelling on. Bigger readers and more training tokens both improve the aggregate and break specific skills. If you only watched the loss curves, you'd ship the 20M R=4 checkpoint and wonder why your code got worse.

Deployment-wise, the design is pragmatic. The GGUF contains the backbone plus the trained reader and arbitration tensors, while the 51B PLE stays external as a quantized sidecar file. The inference path lives in a llama.cpp fork, and a separate runtime test on WikiText-2 showed Q8_0 retains 99.1% of the reader's NLL gain. You lose almost nothing running the memory at 8-bit, and the backbone stays at whatever quantization you already use.

Why this is bigger than 5% perplexity

The portable-memory insight is the durable part. The expensive component of Flash-Next's n-gram capability transferred cleanly to a 0.8B backbone, and the transfer cost was a tiny reader trained for 15M tokens on free GPUs. That flips the usual assumption that capability lives in the backbone and scales with parameter count. For local deployment, it suggests a cheaper path: keep a medium backbone, graft a frozen memory from a bigger model, and spend your training budget on the interface between them.

The caveats are equally important. Aggregate metrics lied twice in this experiment: once when the 20M reader looked better than the 15M one, and again when R=4 looked better than R=1. The evaluation that mattered sliced by skill, not by sum. Anyone planning to reproduce this should build the skill-sliced eval harness first.

The natural next step is the 35B-A3B MoE. The author wants to run the same graft at that scale, and the 0.8B study is now a much clearer recipe for reader size, memory placement, and gating. It needs real compute, though. Free Kaggle notebooks stop being sufficient somewhere between 0.8B and 35B.

Common pitfalls

The 5.05% reduction is validation perplexity, and the author says so explicitly. It is not a benchmark accuracy claim. Worse, the 20M-token reader improved aggregate loss while regressing on math. Slice your evals by skill before shipping anything built this way.

Q2_0 needs 37.6GB across RAM and VRAM. The actual requirement is a small GPU plus a large, fast system RAM pool; on a 12GB card alone the model doesn't fit. DDR5 at 5600MT/s is load-bearing in the Strata numbers, and on slower or single-channel setups the offload path becomes the bottleneck.

Both projects ship modified runtimes. The Qwengram GGUF needs the llama.cpp fork to read the PLE sidecar, and Strata is its own CUDA engine. Downloading just the model file gets you a load error and a confusing afternoon.

Leave thinking mode off for routine tasks. The 27B rambling post resonated because everyone has hit this: a reasoning model will burn a thousand tokens psyching itself up to run a test suite, and at single-digit tok/s that's minutes of waiting. Enable thinking for hard reasoning, disable it for everything else.

Don't assume quantization retention carries over. Q8_0 kept 99.1% of the reader's NLL gain, but that was measured at Q8_0. The backbone, the gate tensors, and the PLE tables each respond differently to lower bit depths, and the gate is the one you least want to damage. Re-evaluate at your target quant instead of extrapolating.

One thing to remember

The weights are becoming a scaffold. Strata changed nothing about the model and got 4x the throughput. Qwengram changed nothing about the backbone and cut validation perplexity by 5%. Both projects treat the model file as the fixed part of the system and spend their effort on what surrounds it: orchestration, memory sidecars, gating, quantization placement. If you're planning a local Qwen deployment, plan for an assembly, not a download.

The Bottom Line

If you're on a 12GB card with 64GB of fast system RAM and want the fastest usable Qwen locally, use the Strata engine at Q2_0 or IQ2_XS. You get 65.1 or 52 tok/s output and 543 or 472 tok/s prompt processing for 37.6GB or 39.2GB combined, and the 4x prompt-processing gain over llama.cpp is what makes large-context workflows practical.

If you're shipping a small on-device model and can't afford backbone fine-tuning, graft a frozen and quantized PLE with an R=1 reader at layers 3 and 9 plus a token-dependent gate. You get 5.05% lower validation perplexity, 99.1% of the gain survives Q8_0, and the training runs on free GPUs. Build the skill-sliced eval first, because aggregate loss will mislead you.

One thing to watch: the 35B-A3B graft. The author of Qwengram is planning it, and the recipe from the 0.8B study is much clearer than it was a week ago. If the PLE transfer holds at that scale without the math regression, expect "medium MoE plus giant frozen memory" to become the default template for local reasoning models within the next few months.