The post-training bottleneck

Distillation is how most production LLMs actually get built. You take a powerful teacher, transfer its behavior into a student small enough to serve at reasonable cost, and ship. But the student inherits more than the teacher's skills. It inherits the teacher's mistakes.

A current batch of post-training work, three papers and a tooling release, attacks different corners of the same problem. The supervision that flows from teacher to student is noisy, biased, or mistimed. Fix that, and the capacity transfer mostly takes care of itself.

Coupled Calibration and Learning (CCL) handles the case where the teacher is systematically wrong on your target domain. γOPD fixes how credit is assigned across tokens during on-policy distillation, then mixes in real outcome reward so the student can outgrow the teacher. SKIP keeps reasoning models from overthinking without wrecking accuracy. Soup removes the hours spent fighting training infrastructure, which is where a third to half of team time goes.

The common thread: post-training is a supervision problem, not a capacity problem.

Key numbers

• 3.32 GB peak VRAM to fine-tune Llama-3.1-8B-Instruct with layer streaming and NF4 quantization, bit-exact against a normally resident run. That is a gaming laptop, not a cluster.
• 119.6 tok/s training throughput on an RTX 3050 Laptop, reproduced at 113.0 tok/s on an H100 in the same memory budget.
• 30-50% of team time lost to infrastructure before the tooling got fixed.
• Polynomial convergence for CCL's student toward the best policy its architecture can express, with zero target-domain reward labels.

Teacher bias is the silent killer

The setup for CCL is the one every distillation team eventually hits. Your teacher handles source questions well, you have reward feedback there, and you have none on the target domain where you actually deploy. Under covariate shift, the teacher's reliability on target questions is unknown. Direct imitation copies whatever the teacher does, bias included, because every token looks like equally valid ground truth.

CCL stops treating the teacher as a fixed oracle. Each iteration recalibrates the teacher using source feedback, then uses the calibrated teacher to train the student on target questions. The updated student informs the next calibration round. Teacher and student bootstrap each other, with the source reward anchoring the loop.

The convergence guarantee is stronger than the usual distillation story. The student's expected average KL divergence to the oracle student converges to zero at a polynomial rate in the number of iterations. "Oracle student" means the best policy within the student class under reference-regularized target reward, not the unrestricted optimal policy, so the guarantee is honest about the student's capacity limits. There is also a separation result: regularized direct matching can stay bounded away from that oracle even when the teacher's regularized target reward beats every student policy. A teacher that is outright better than your entire student family can still poison naive imitation. CCL recovers the best student anyway.

Practical translation: if you are distilling into a smaller model for a domain where you lack reward labels, you no longer have to choose between trusting a biased teacher and waiting until you can label target data. The calibration loop gets you the best student your architecture can express, using only feedback you already have.

The credit assignment trade-off nobody escapes

On-policy distillation has a long-running trade-off. Token-level objectives are stable but local: each token gets feedback based on what the teacher would do right there. Sequence-level objectives capture future consequences, like a real RL return, but their variance grows with the horizon, which punishes long reasoning traces.

The γOPD paper shows these two are not as separate as they look. Practical token-level OPD can be interpreted as a temporal approximation to the sequence-level reverse-KL gradient. Once you see them as the ends of a spectrum, the fix is obvious: discount the credit. γOPD scales future tokens' contribution by a discount factor γ, a dial between myopic stability and full-horizon fidelity. Its variance bound is horizon-independent, meaning longer sequences no longer force you to shrink batch size or accumulate gradients to keep training stable.

Objective	Supervision granularity	Credit horizon	Variance behavior	Uses outcome reward
Token-level OPD	Per-token	Myopic	Low, stable	No
Sequence-level OPD	Whole-sequence	Full horizon	Grows with horizon	No
γOPD + RBM	Discounted temporal	Tunable with γ	Bounded, horizon-independent	Yes

The reward-compatible bounded mixing (RBM) mechanism matters too. Pure on-policy distillation is teacher-dependent by construction. RBM mixes verifiable outcome feedback into the discounted OPD advantage, so the student can outgrow the teacher's opinions. On math and code reasoning, the authors report consistent gains over existing OPD methods across vanilla distillation, size-mismatched teacher-student pairs, and multi-teacher setups.

Quick Take: most distillation failures come from supervision being noisy, biased, or mistimed, not from the student being too small.

The tooling gap

All of this assumes you can actually run the experiment. Soup's pitch is blunt: experienced teams spend 30-50% of their time fighting infrastructure instead of improving models. The fix is one YAML config, no SSH, and automatic batch size, GPU detection, and quantization.

The headline trick is layer streaming. The frozen base model stays out of VRAM entirely and feeds the GPU one decoder layer at a time. That is how an 8B model trains on a 4 GB laptop GPU: Llama-3.1-8B-Instruct with NF4 quantization measured 119.6 tok/s at 3.32 GB peak on an RTX 3050 Laptop, bit-exact against a normally resident run. An independent reproduction on an H100 measured 113.0 tok/s in the same 3.32 GB budget. Layer streaming is opt-in and still marked beta. The docs flag caveats in the same measured tone: the throughput figure predates a correctness repair that cost 4.8% at 32B, and it has not been re-run on a 4 GB card since. There is also a free Colab T4 script that caps the process at 4 GB and asserts bit-identical weights, which is exactly how you verify a memory trick like layer streaming.

The workflow is the point. pipx install "soup-cli[train]", soup init --template chat, soup train. One command merges the LoRA, another exports GGUF for Ollama, another serves an OpenAI-compatible endpoint. Double quotes matter: "soup-cli[train]" is the only spelling that works in cmd.exe, PowerShell, bash, and zsh. If a tutorial used single quotes and pip rejected it, that is why.

What the community is saying. The v0.75.0 release notes read like a catalog of every way a training tool can quietly lie to you. When I re-ran an old MLX config, six validated settings, train_on_responses_only, warmup_ratio, scheduler, weight_decay, optimizer, and max_grad_norm, had been silently dropped by that backend. Only 8 of 32 optimizer names have an MLX equivalent; the other 24 are refused by name instead of silently becoming AdamW. A typo like quantizaton used to validate clean, then the run proceeded without the setting applied. v0.75 fails the load instead, names the field you probably meant, and exits 1. Validation loss used to be computed on every backend and thrown away, no metrics column, no event field. Now it is recorded, streamed, and displayed, and soup doctor --config lists what a backend does not read.

The change that bites people with recorded baselines is grpo_variant: gspo. The published sequence-level objective replaced a column-centering heuristic in which a padding token shifted the gradient of every row sharing its column. Existing gspo configs cannot reproduce prior runs, so old numbers are not comparable with new ones. Not a bug. Just re-run.

Concise reasoning without the accuracy cliff

Reasoning models have a habit of overthinking. Chain-of-thought works until it doesn't: longer outputs cost compute and latency, and can degrade accuracy outright. The naive fix, penalizing length, compresses the output but also compresses the reasoning itself.

SKIP takes a different route. A lightweight SFT pass first adjusts the model's output style toward concision. Then a knowledge probing mechanism prompts the model to commit to an answer at each reasoning step. Those intermediate commits become the training signal. Steps that led to a correct final answer become positive examples; steps that misled become negatives. The resulting pairs feed a DPO step, so the model learns which reasoning steps are worth keeping and which are elaborate detours.

Reported results line up with the intuition: better compression than length-penalty baselines, with the accuracy loss largely mitigated. The gains transfer to out-of-distribution datasets, and the ablations confirm that both the probing and the step-level preferences are load-bearing. Reward the model for stopping when it already knows the answer, rather than punishing it for thinking.

Common pitfalls

The first trap is trusting teacher outputs as clean labels under covariate shift. Benchmarks on source questions look fine while the student quietly inherits systematic errors on the target domain. If target reward labels are unavailable, calibrate the teacher with source feedback as you train, the way CCL does, instead of freezing it and copying outputs.

The second is picking one extreme in the credit assignment trade-off. Token-level OPD trains stably but the student never learns from future consequences. Full sequence-level objectives capture the future, but variance grows with reasoning length and long traces destabilize training. Hand-rolled distillation loops should add a discount over future tokens from the start; γOPD exists because both extremes mislead.

The third is trusting a config that a backend half-reads. A YAML key that validates and then gets ignored by the MLX or transformers backend corrupts the experiment silently. My team now diffs the effective config against the file we wrote before trusting any run longer than an hour, and we treat unknown config keys as load failures. When a tool only warns, assume the warning is about the setting you care about.

The fourth is compressing reasoning by penalizing length. Length penalties punish correct-but-verbose reasoning as hard as actual overthinking. Step-level supervision, with preference pairs built from intermediate correctness, gives the model a signal about which tokens were wasted and which carried the argument.

The fifth is assuming version compatibility in the training stack. torch 2.5.1 with TRL 0.29 and newer cannot import, so every preference trainer dies. Python 3.13+ resolves PyTorch wheels that crash in the native extension before training starts. Pin torch to 2.6.0 or newer and stay on Python 3.10-3.12 until the wheel situation is validated. If you recorded baselines with the old gspo heuristic, re-run them; the corrected objective will not reproduce those numbers.

One thing to remember

The pattern across all four pieces of work is the same. The teacher's output gets treated as ground truth, and every failure mode in this article, teacher bias, myopic credit, overthinking, silently dropped config options, comes from trusting that signal too much. Every fix adds a second opinion: source reward for calibration, discounted future credit, step-level correctness, or a config that fails loudly. Build your post-training pipeline around that and most of these problems stop being silent.

The bottom line

If you are distilling into a smaller model for a domain without reward labels, adopt a calibration loop like CCL, because direct imitation under covariate shift transfers the teacher's systematic bias, and no amount of source-domain data will fix it.

If you are running on-policy distillation on long reasoning traces, skip pure token-level and pure sequence-level objectives and use discounted credit assignment with outcome-reward mixing, because a horizon-independent variance bound is what keeps training stable when traces lengthen.

If you are constrained to a single consumer GPU, use layer streaming instead of waiting for cloud quota, because an 8B model training at 119.6 tok/s in 3.32 GB is enough to run real post-training experiments today. One thing to watch: the objective corrections, gspo in Soup and the temporal-credit unification in γOPD, mean published baselines from earlier this year are shifting. Expect re-runs and re-validations of old recipes within six months.