Three memory problems that look different until you squint

The activation cache, not the weights, is what kills a 70B training run at 8K sequence length. The KV cache, not the model, is what caps your concurrency on a long-context server. And a raw pixel stream, not the entropy model, is what limits a lossless image codec.

Three papers landed with the same instinct: stop approximating the data, restructure it so the exact representation gets smaller. BitSparse compresses the activation cache losslessly by exploiting a property that fell out of favor in LLMs for five years and is now coming back. DropKV turns an NP-hard eviction problem into independent per-token scoring with a constant-factor guarantee. WaveLLM decomposes images into wavelet subbands so its LLM entropy model has less to learn at each prediction.

All three are memory work. None of them buy the savings with accuracy.

BitSparse packs away the sign bits ReLU never uses

ReLU went out of fashion for good reasons. SwiGLU and GeLU gave smoother gradients and better loss curves, and the sparsity ReLU produced was mostly wasted anyway because dense GPU kernels can't skip zeros cheaply. The return of ReLU and ReLU² in sparse feed-forward designs changes the arithmetic. If a large share of your activations are exact zeros, and every nonzero value shares a sign, you have a compressible tensor sitting in HBM for the entire backward pass.

BitSparse's first observation costs nothing to state. A ReLU output is never negative, so the sign bit of every bf16 value in the cache is zero. Left interleaved with the exponent and mantissa, those bits are dead weight. Packed into their own stream, they cost almost nothing, which is the optional sign-bit packing that pushes the format toward the theoretical entropy floor for the tensor.

The reported numbers: 31% lower per-token VRAM on Modded-NanoGPT and 30% on Nemotron-H, with step-time overhead under 1% at practical sequence lengths. A 31% cut in per-token memory stretches the same budget across roughly 45% more tokens. That's the difference between a 4K-context batch fitting and not fitting.

The implementation is the part that matters for adoption. Compression and reconstruction run as custom Triton kernels fused into an autograd function, so the backward pass gets the exact tensor back without a separate decompression stage and without an extra round trip through HBM. A preallocated buffer pool handles variable-size tensors, which avoids the host synchronization that would otherwise show up as a stall every step and quietly eat the savings.

Quick take: BitSparse is the rare memory optimization that costs almost nothing to adopt, because it's lossless, it's fused into autograd, and the activation pattern it depends on is exactly the one sparse FFN designs are already moving toward.

DropKV scores tokens one at a time and gets a guarantee

The KV cache grows linearly with context, and every eviction method has to answer the same question: which token's removal perturbs the output least? Stated exactly, that's a combinatorial problem over subsets. It's NP-hard. So everyone uses heuristics.

Attention-score methods came first and are still the default. Value-aware methods improved output quality by looking at what each token contributes to the residual stream rather than how much attention it attracted. But the better heuristics got, the harder they were to write as kernels, and a scoring rule that can't be fused is a scoring rule that loses its speed advantage at the exact moment you need it.

DropKV decouples the decision. Instead of choosing a set jointly, it scores each token independently by how much removing it would perturb the residual output. The combinatorics disappear. What remains is a per-token ranking that admits a constant-factor approximation guarantee under a cone condition the authors check empirically on three long-context models. A guarantee that holds under a verified condition is weaker than an unconditional one, but it's more than the alternatives offer.

The implementation numbers are the interesting part. Fused Triton kernels avoid materializing the full attention matrix, which at 128K context is the difference between an O(n²) allocation and a streaming score. That gives a 19.8x speedup on the scoring kernel, which translates to a 9.9% end-to-end prefill speedup at matched batch size. Prefill is the phase where the KV cache is being built, so shaving a tenth off it is worth chasing on a serving cluster. On RULER and LongBench, DropKV holds the lowest inference latency among eviction methods at equivalent cache budgets, and it stays faster than the dynamic-cache baseline.

WaveLLM changes the input before it changes the model

Lossless compression is entropy modeling all the way down. Get the probability distribution right and the arithmetic coder does the rest. Recent work plugged LLMs into that loop and got better predictions, but most of it serializes the image into a pixel or residual sequence and lets one autoregressive stream handle everything. Flat sky, sharp edges, textured fabric and fine diagonal detail all share a single prediction context.

WaveLLM's argument is that model capacity isn't the only lever. The predictability of the input representation matters just as much. Split an image into multi-level wavelet subbands and a heterogeneous prediction problem becomes a set of homogeneous ones. The low-frequency approximation band is smooth and easy. The diagonal detail bands are sparse and noisy in a different way. An autoregressive model that sees one subband at a time has a much narrower distribution to learn.

The paper reports 2.51, 2.24 and 2.27 bits per subpixel on Kodak, CLIC.pro and DIV2K. Against a PixelCNN codec using the same wavelet transform, that's a gain of 0.43, 0.36 and 0.46 bpsp. The Kodak gap works out to roughly 15% fewer bits from swapping the entropy model while leaving the transform untouched. Same subbands, better predictor. That's the cleanest ablation you could ask for.

The model is Llama-3.2-3B with LoRA adaptation, and the size is a design constraint rather than a limitation. A lossless codec needs the same model on the encoder and the decoder, so every parameter ships twice. Three billion parameters runs on a single consumer GPU. A 70B entropy model would compress beautifully and be impossible to deploy.

Getting an LLM to predict wavelet coefficients takes a numerical interface: value embeddings to map integer coefficients into text token space, and a coefficient prediction head to map back out. A task-adaptive prompting strategy lets one shared model handle the different subbands instead of training a separate head per band.

The three techniques side by side

Technique	What it shrinks	Mechanism	Reported gain	What it costs
BitSparse	Activation cache during training	Lossless bit-packing of sparse ReLU outputs in fused Triton kernels	31% per-token VRAM on Modded-NanoGPT, 30% on Nemotron-H	Under 1% step overhead at practical sequence lengths
DropKV	KV cache during inference	Decoupled per-token residual-output scoring with a cone-condition guarantee	19.8x scoring kernel, 9.9% end-to-end prefill speedup	An extra scoring pass at each eviction step
WaveLLM	Image bitstream	Multi-level wavelet subbands plus an LLM entropy model with a numeric interface	2.51 bpsp on Kodak against 2.94 for the same-wavelet PixelCNN	A 3B model in the encode and decode loop

Key numbers

31% and 30%: per-token VRAM reduction from BitSparse on Modded-NanoGPT and Nemotron-H. 1%: ceiling on BitSparse step-time overhead at practical sequence lengths. 19.8x: scoring-kernel speedup from DropKV's fused Triton implementation. 9.9%: end-to-end prefill speedup at matched batch size. 2.51 bpsp: WaveLLM on Kodak, against 2.94 for a PixelCNN codec on the same transform.

What trips people up

The first mistake is reaching for activation compression on a model that has nothing to compress. BitSparse works because ReLU and ReLU² produce large numbers of exact zeros and non-negative outputs. Run the same bit-packing over GeLU or SwiGLU activations and you'll find the zero runs aren't there, the sign bits are doing real work, and the format expands rather than shrinks.

The second is assuming sign-bit packing is safe for any layer. It isn't. The moment you apply it to a tensor that can go negative, you corrupt the backward pass silently, and the loss curve won't tell you until several thousand steps later. BitSparse is correct because the ReLU guarantees the property, not because the format is universally applicable.

The third is treating the buffer pool as an optimization detail. Allocating a fresh buffer for every compressed tensor of variable size forces host synchronization, and that sync can cost more than the compression saves. If you reimplement the format without the preallocated pool, expect a regression and expect to blame the kernels.

The fourth is scoring eviction on attention weights alone. DropKV's whole point is that output perturbation depends on the value vectors, not just the attention mass. Tokens that look important in the attention map can contribute almost nothing to the residual stream, and evicting them is free. The reverse is also true, which is where attention heuristics lose quality.

The fifth is wavelet decomposition without matching the entropy model to subband statistics. If you decompose the image and then feed every subband through the same prompt and the same context, you've re-entangled the problem you just separated. The per-band adaptation is where the gain lives.

One thing to remember

Every one of these papers wins by finding structure that already exists in the data and making the representation match it. ReLU outputs are non-negative, so the sign bits are free. Residual-output perturbation decomposes per token, so the combinatorics were never necessary. Wavelet subbands are statistically homogeneous, so the entropy model doesn't need to be a generalist. In each case, the compression ratio came from the shape of the signal, not from a bigger model or a smarter approximation.

When to reach for each

If you're training a model with ReLU-family activations and long sequences, adopt BitSparse-style lossless activation packing now. The savings are free, the overhead is under 1%, and the format is exact, so there's no accuracy question to argue about.

If you're serving long context on a tight cache budget, drop attention-mass eviction and move to decoupled per-token scoring. You get a better quality-per-byte tradeoff at the same cache size and you get it with lower latency, which is the combination attention heuristics never delivered.

If you're building a lossless image codec, spend your effort on the input representation before you spend it on model scale. WaveLLM's 0.43 bpsp gain on Kodak came from the transform-model pairing, not from parameter count, and a 3B model is what makes the result shippable. Watch this space closely: representation-first codecs are the direction, and expect the next round of results to push the same idea into video.