Last month someone picked up an unopened datacenter GPU on Craigslist for $4,000 and walked away planning to host their own models. A few weeks later, a 44M-parameter LLM started answering questions from inside a talking Peppa Pig plush toy, fully offline. Different projects, same driver: people want capable models on hardware they own, and a growing chunk of research is organized around making that work.

The papers and releases from the last month share a thesis. Low-rank compression, low-bit attention, tokenizer design, and tiny quantized models are all converging on one idea: compressing a model is a systems problem. The parts interact. Optimize them in isolation and you leave most of the quality on the table.

Per-matrix optimality is a trap

Truncated SVD is the default starting point for low-rank compression, and for good reason: for a single matrix, it's the best low-rank approximation you can get. That's the Eckart-Young theorem. The problem is that a Transformer isn't a single matrix. Compress every matrix independently, and the individual truncation errors compound through the block's nonlinear forward pass. Eckart-Young optimality doesn't survive contact with the model.

A new paper (arXiv 2609.15838) replaces per-matrix compression with a three-level chain, inspired in part by hierarchical variational optimization in quantum many-body methods. L1 runs whitened SVD per matrix. L2 jointly optimizes whole Transformer blocks, which is where the compounding errors get corrected. L3 refines against the full language-modeling loss, end to end. Everything runs on 256 calibration sequences, no instruction data, no recovery data.

On LLaMA-7B at 60% compression, the chain takes WikiText-2 perplexity from 42.1 to 19.1 to 11.4.

Stage	Optimization scope	Method	WikiText-2 PPL
L1	individual matrices	whitened SVD truncation	42.1
L2	Transformer block	joint block optimization	19.1
L3	full model	end-to-end LM loss refinement	11.4

Most of the gain lives in the block-level stage, and that stage doubles as a regularizer. Skip it and Penn Treebank perplexity gets 24 points worse. The authors tried closing that gap with extra end-to-end training and couldn't. You can't fine-tune your way out of a bad intermediate representation.

The results hold across 20-80% compression and five architectures up to 13B parameters, on in-distribution and out-of-distribution benchmarks. The authors are careful about what they claim: cross-architecture rows used architecture-specific configurations, the ratio sweep wasn't run under one common protocol, and downstream accuracy stays below the dense model. This improves compression fidelity, but it isn't a replacement for the original.

The KLD debate got spicy

Quantization is having its own argument about how to measure success. ByteShape released ShapeLearn quants for Qwen 3.8 27B, claiming the 3.84 bpw model hits 99.63% of the BF16 aggregate across eight benchmarks, with the 3.23 bpw variant at 98.72%. Their inference speedups look legit: DFlash2 delivers 1.34-2.10x baseline throughput, and multi-token prediction adds another 1.28-1.66x with temperature sampling. Lower bit width translates directly into tokens per second on consumer GPUs.

The argument is about KLD. Unsloth's Dynamic V3 quant had roughly 20% lower KL divergence than ByteShape's comparable model, which looked like a win for the other camp. Then the task results came in: the lower-KLD model scored 95.55% of BF16 across the benchmarks, the ShapeLearn model 97.33%. KLD is not a quantization leaderboard. It flags a quant that has fallen off the cliff, and it doesn't tell you which quant is better than another. ByteShape's paper on this was accepted to the EMNLP 2026 Industry Track, so the argument is heading for the literature.

When I re-benchmarked my own quants after the Dynamic v3 release, I kept getting the same result: the model with the closest token distribution wasn't the one winning GSM8K or IFEval. Measure tasks.

Also this month, Voodoo Quant went MIT-licensed. Voodoo runs every candidate quant level for every tensor at the same time, freezes all candidate weights, and trains a scalar gate per tensor per quant level with gradient descent. Softmax keeps every quant level receiving gradient even as an annealed tau pushes each gate toward one choice. The loss balances KLD against the BF16 reference with a target filesize. The author's own testing says Unsloth's Dynamic 3.0 wins at mid-to-high quant levels and Voodoo wins at the aggressive low-bit end. Using a backwards pass to pick per-tensor quant layouts is a clear departure from static analysis, and now anyone can build on it.

Quick Take: Two independent threads, low-rank compression and quantization, are landing on the same conclusion. The interaction between parts matters more than any single per-tensor choice.

Attention kernels are the next frontier

Weights aren't the only thing worth compressing. VC-Attention (arXiv 2609.15810) targets the attention computation in diffusion transformers, where long spatiotemporal sequences make attention the dominant deployment cost.

Two things block low-bit attention kernels. First, value outliers. A block's quantization scale is set by its largest entries, which leaves typical entries crammed into a narrow representable range. Value outliers follow no fixed channel or spatiotemporal structure, so the smoothing tricks that work for queries and keys don't help. Second, softmax. Low-bit Tensor Cores accelerate the two matrix multiplications, but the high-precision exponential between them becomes the longest pipeline stage on datacenter GPUs.

VC-Attention pairs value smoothing with a fused probability cast. V-Smooth reorders value tokens using lightweight online clustering so the tokens in a hardware block quantize well together, then quantizes the residual after subtracting the block mean and restores the mean from the row sum the online softmax already maintains. ExpCast-FP8 maps log-domain scores directly to E4M3 probability codes with one fused multiply-add. No FP32 exponential, no format conversion.

Across Wan2.2, LongCat-Video, HunyuanVideo-1.5, and MiniMax-H3, the kernel runs 1.46-1.59x faster than BF16 FlashAttention-4 on datacenter Blackwell and Hopper, and 2.3-3.6x faster on workstation cards. End-to-end clip generation is 1.13-1.19x faster on datacenter parts and 1.36-1.70x faster on workstations. Those multipliers look small until you price a cluster running video diffusion all day. And the whole thing is training-free: drop it into an existing deployment, no weights touched.

Tokenizers were doing compression all along

Multilingual models hide a cost that most people never measure: the shared vocabulary. One tokenizer for many languages means uneven compression across them, and the embedding and output matrices grow with every added language. Small models feel this directly, and most of it is dead weight when you only serve a subset of the languages.

The modular tokenizer framework (arXiv 2609.15528) learns large modular BPE and Unigram tokenizers, then extracts subtokenizers tailored to any language subset. Pretraining samples subtokenizers to form batches and restricts predictions to the active vocabulary subset, so training stays efficient despite a big vocabulary. At inference you load only the vocabularies you need, which reduces memory and speeds up generation without hurting performance, and the subtokenizers hit compression on par with monolingual tokenizers.

For a small model serving one or two languages, this can shrink a real fraction of the embedding parameters. Least glamorous compression, most practical kind.

What 20 MB gets you

SHADOW-50M is the case for treating compression as a design constraint rather than a post-training step. It's 44M parameters trained from scratch on 45B tokens, with ternary {-1,0,+1} weights. The complete model ships in 19.8 MB and runs at about 1,900 tok/s on a laptop CPU using roughly 41 MB of RAM. The same 159 KB kernel compiled to WebAssembly runs in a browser tab at about 500 tok/s. That's fast enough for real-time conversation on a mid-range laptop, no GPU and no cloud.

Two choices make it possible. The vocabulary is 73,880 tokens represented as fixed 512-bit fingerprints in a 4.7 MB frozen table, no trained embedding. And computation the model can't reliably do, arithmetic, percentages, dates, weekdays, units, is handled by fixed circuits at the readout. When SHADOW decides something needs calculation, it writes something like [calc]347*86[eq] into its own token stream and the circuit fills in the correct digits. No calculator API, no tool call, nothing pasted back into the prompt.

Key numbers for SHADOW-50M: 44M parameters, 19.8 MB total model size, 1,900 tok/s on a laptop CPU, 41 MB RAM at runtime, 73,880-token vocabulary in a 4.7 MB frozen table, 0.03 ms to load a stored attention state from disk, 500 tok/s in a browser via WebAssembly.

The frozen table turned out to be more than a size trick. The first release was missing about 8,600 English word pieces, so "fitzgerald" could only reach the model as "fitz". Every fine-tuning run that learned the missing words broke something else. The fix was adding the 8,600 missing rows directly to the frozen fingerprint table. No training, same weights, all 34 previously published answers unchanged, and the model could suddenly read the new pieces. A trained embedding can't accept thousands of new rows without training. A frozen table can.

The long-term memory design is the most unusual part. When SHADOW stores a record, it reads the text once and writes its attention state to disk at 1 bit, 288 bytes per token. A query like the condition of Patient P-204 hits an index that finds the record in roughly a microsecond, and the stored attention state loads back into the model in about 0.03 ms. The text is never re-read. At 100M tokens, that's a 28.8 GB archive plus a 2.2 GB index on disk, with the process using about 28 MB of RAM because everything is memory-mapped and a question only touches the pages it needs. The index reinforces records that get used, and top-1 retrieval improved from 0.571 to 0.743 without any training.

The author published the comparison against Supra-50M-Reasoning, a bf16 Llama-style model at 51.8M parameters, and didn't hide the bad news.

Supra beats SHADOW everywhere: ARC-Easy 0.435 vs 0.307, PIQA 0.600 vs 0.570, WikiText-2 perplexity 165 vs 186. The qualitative runs tell a different story. Asked for a joke, SHADOW answers with the computer going to the doctor; Supra thinks for 189 tokens about what computers are and stops. Asked for 15% of 240, SHADOW says 36 dollars; Supra says 250, then corrects itself to 150. Asked for the condition of a stored patient record, SHADOW quotes it from disk; Supra, with the records in its prompt, produces a sentence about allergic reactions. The benchmarks aren't measuring what SHADOW was built to do.

The quantization floor shows up clearly in the same comparison. Supra is 103.6 MB in bf16, 56.2 MB at 8-bit, and quality collapses below that: int4 pushes perplexity from 165 to 193, and ternary breaks it entirely. SHADOW lives at ternary because it was trained there, not quantized into it. Post-training quantization has a floor, and it sits well above where a model was born.

The community response went somewhere nobody expected. A contributor sent a pull request with a CUDA engine, a quantization tutorial, and a talking Peppa Pig plush toy running SHADOW inside it: microphone, small speaker, ~$35 board, fully local. Everything is MIT licensed, code and weights included. The maintainer hasn't merged the ~11,000 lines yet because verifying that much CUDA by hand is a heavy ask. But the demo exists, and the model inside the toy is 20 MB.

The local runtime stack is catching up

All of this needs software that can use it. KoboldCpp v1.121 is a snapshot of how fast local runtimes are absorbing these techniques. The release adds MiniMax H3 media references, so you can attach audio clips and reference images and generate video entirely in-process. The release notes open with a music video generated inside KoboldCpp itself, in seven-second segments. Video LoRA support, grammar memoization fixes, tool-calling fixes for Kimi and DeepSeek models, an optimized adaptive-P sampler, runtime LoRA selection in the image-generation UI. It's a long changelog, and it's typical of how this project ships.

What the community is saying: the v1.121 thread reads like a changelog of friction removed. Streaming race conditions gone, UTF-8 handling fixed in WebSearch, segfaults on model-load failure prevented. I've hit every one of these bug classes at some point, and watching them get fixed at repo-release speed is what makes local-first workflows feel dependable instead of experimental.

Hardware is showing up to run all of it. The Craigslist GPU from the top of this piece, unopened at $4,000 against a $5-6k street price, is a bet that local hosting is worth real money right now. The buyer verified the seller on LinkedIn first, which is the correct amount of paranoia for that market.

Common pitfalls

Compressing each matrix as if the others don't exist. Eckart-Young says per-matrix SVD is optimal, and the three-level chain paper shows 23 points of perplexity disappear once you optimize at the block level. The errors compound through the forward pass. Optimize at the block level, or don't bother.

Using KLD as your quality metric. It tells you when a quant has fallen off a cliff, not which quant is better. The ByteShape results show a model with 20% lower KLD scoring 95.55% against 97.33% on real tasks. Measure tasks.

Quantizing values the same way as queries and keys. Value outliers follow no channel structure, so per-channel smoothing misses them entirely. VC-Attention needed a separate value-smoothing pass precisely because of this.

Assuming you can quantize down to any bit width. Supra-50M loses quality at int4 and dies at ternary. SHADOW works at ternary because it was trained that way from the start. If you need extreme compression, train for it.

Fixing tokenizer gaps with fine-tuning. Every fine-tune that learned the missing word pieces broke something else. Adding rows to the frozen fingerprint table fixed it with zero training, because a fixed lookup table can grow where a trained embedding can't.

One thing to remember

Every result in this cluster points the same direction: compression quality comes from the design of the system, not the choice of a single technique. The block optimizer matters more than the matrix factorization. The per-tensor quant layout matters more than the bit width. The tokenizer and the memory architecture matter as much as the weights. The wins live in the interactions.

The bottom line

If you're compressing a model with low-rank methods, adopt block-level joint optimization as your baseline. Per-matrix SVD leaves most of the quality on the table, and the block stage is what makes end-to-end refinement work.

If you're quantizing for consumer GPUs, pick quants by task benchmarks, not KLD, and don't chase aggressive bit widths on models that weren't trained for them. At the extreme end, a tiny ternary model with external disk memory already beats a fragile int4 version of a larger model on narrow, grounded tasks.

One thing to watch: low-bit attention is the next wave. VC-Attention's fused softmax cast is training-free