Pretraining sets the ceiling. Everything after it decides how much of that ceiling you actually reach: the fine-tuning data you choose, the knowledge you transfer, the cost you pay per token. Four recent results, three papers and one very public home-lab experiment, show how much of the action has moved to the post-training lifecycle. One gives a theory for why some concepts never learn during fine-tuning. Another builds a statistical test to catch models trained on someone else's outputs. A third distills a teacher into plain text instead of weights. The last one converts dense models into sparse MoEs on an 8GB GPU.

The prior barrier: why some concepts won't learn

Supervised fine-tuning has a failure mode that shows up long before you look at the loss curve. Some concepts the model picks up immediately; others never land. The usual explanation is data scarcity: you didn't show enough examples. The new paper argues the problem is upstream, in the pretrained weights themselves.

The paper introduces a quantity called the prior barrier: how strongly the pretrained model supports competing concepts over the target concept. If the base model already leans toward a competing answer, the barrier is high, and the SFT signal has to push against a strong prior. If the model is roughly neutral, the barrier is low. Think of it as the model's prior confidence in the wrong answer relative to the right one.

Two findings matter here. First, prior barriers follow a long-tail distribution. A few concepts in the head have low barriers and learn quickly. The long tail of concepts has high barriers, and each one needs substantially more supervision. Second, the paper derives a predictive risk bound that couples the barrier with accumulated SFT evidence. The practical reading is blunt: head concepts hit diminishing returns fast, while tail concepts stay under-trained no matter how many total examples you throw at the problem, unless the selection actively favors them.

PASS: spending the instruction budget where it counts

PASS is the method that falls out of that analysis. It's an adaptive instruction selection algorithm that repeats two steps per iteration. It constructs reference-derived concepts, then estimates how much distinguishing evidence each candidate instruction provides for each concept. The selection budget flows toward concepts whose coverage is still insufficient under the current selection.

What this buys you is a selection policy that asks two questions at once: does this instruction give useful evidence, and does this concept actually need it? Uniform sampling only answers the second question by accident. The paper reports consistent wins over seven state-of-the-art instruction selection methods across four backbone-budget settings, and the ablation shows that adaptive allocation beats uniform allocation with the same total budget.

The practical upshot: if you ever select SFT data by random sampling or by a fixed difficulty score, you are probably over-investing in the head of your concept distribution and starving the tail that drives most of your task failures.

Quick Take: Post-training is turning into a measurement problem, deciding which concepts need evidence, detecting whose outputs trained a suspect model, and measuring which layers survive being made sparse.

Distillation attribution: catching a stolen model

Unauthorized distillation, training a model on a proprietary model's outputs, is a growing threat to API providers. The threat model is straightforward: someone queries your model heavily, saves the responses, and fine-tunes a cheaper competitor on them. The open question has always been attribution. Can you look at a suspect model and determine whether it was distilled from yours, or trained independently?

This paper treats the problem as a hypothesis test. The auditor trains two families of shadow models. Distilled shadows learn from the teacher's reasoning traces. Independent shadows learn only from reference answers. That gives you the expected behavior under each hypothesis. The auditor measures how closely the suspect predicts the teacher's reasoning outputs, then uses the shadow models to convert the suspect's score into a calibrated p-value. If the suspect looks far more like a distilled shadow than an independent one, you flag it.

The numbers are strong for a preliminary study. With Qwen2.5-7B as the teacher and Llama-3.2-3B suspects, the test achieves a true positive rate of 1.0 at a significance level of 0.02. Every distilled suspect in the study was flagged, with the false-positive budget held at 2%.

The model sizes matter here. A 7B teacher and 3B suspects are realistic, runnable models, not toy configurations. The expensive part is training the shadow models, which is a one-time cost per teacher-suspect setup. The audit itself is just forward passes.

Key numbers1.0 true positive rate for flagging distilled suspects at a 0.02 significance level 7B teacher, 3B suspects, the configuration validated in the study 2 shadow-model families, distilled and independent, required to calibrate the p-value

Teaching without touching weights: the Primer

Most knowledge distillation assumes you can train the student. That binds the distilled knowledge to a specific architecture and a specific checkpoint, which makes it hard to interpret, hard to reuse, and impossible when the teacher is API-only. Universal Textual Teaching (UTT) removes the assumption entirely. The output of distillation is not weights. It's a natural-language artifact called a Primer.

The pipeline starts with paired evaluations that identify representative gap cases. Then a four-role loop runs. The Student attempts each task. The Prompter turns evaluation feedback into a teaching instruction. The Teacher provides a targeted demonstration. The Synthesizer consolidates validated lessons into the Primer. No parameters are updated anywhere. The Primer is the distilled model, and it can travel to any other student.

The results are hard to argue with. On KernelBench, UTT raises the Student's accuracy from 9.4% to 48.6%, and Fast1 from 9% to 35%. On Omni-MATH-2, mathematical reasoning accuracy climbs from 27.6% to 51.7%. The method also beats representative prompt-engineering and parameter-based KD baselines. And a Primer synthesized for one teacher-student pair generalizes to other students that never participated in the synthesis.

A 40-point jump on a code benchmark sounds absurd until you notice the starting point. These students start near zero on KernelBench, so the Primer is doing a lot of the heavy lifting. The more interesting claim is portability: a textual artifact that survives a teacher-student swap, which no weight-based distillation can do.

Dense-to-MoE on an 8GB card

I spent a couple of weeks converting dense models into sparse MoEs with no pretraining from scratch, and the pattern was clear within days: the last layer is hostile territory. The experiment, posted to r/LocalLLaMA, is about as close to kitchen-table upcycling as this field gets. Big labs do this kind of conversion with enormous compute budgets. The question was whether a hobbyist could do it with 8GB of VRAM.

I converted Qwen2.5-0.5B and SmolLM2-360M, both chosen because they're the largest things an RTX 4060 can handle. The layout is 32 experts with top-8 routing plus a shared expert. The final layer in both models, plus SmolLM2's layer 3, resisted conversion badly, so I left them as dense layers and exported everything to Qwen2-MoE format. The trade felt right: a small MLP compute increase in exchange for keeping quality in the layers that mattered most.

The numbers tell the honest story. MLP compute per token drops to about 40% of the original on Qwen and 44% on SmolLM2. WikiText-2 perplexity goes from 14.66 to 19.20 on Qwen, and from 12.94 to 19.11 on SmolLM2. Mean commonsense accuracy drops from 0.467 to 0.408 on Qwen, and from 0.514 to 0.417 on SmolLM2.

There will always be a quality gap here, because the converted model runs at most half of its original MLP compute. At the 0.5B scale the trade barely makes sense: these models are already cheap to serve. The question everyone in the thread kept coming back to is whether the gap closes as the base model gets bigger. I don't have an answer. Larger upcycled models need more training compute than an 8GB card can deliver, and that's the honest limit of this experiment.

Where these four fit together

Each of these results attacks a different stage of the post-training lifecycle. The table pulls them together.

Method	Problem it solves	What it updates	Headline result	Main constraint
PASS	Long-tail concepts resist SFT	Instruction selection budget, not weights	Beats 7 SOTA selection methods across 4 backbone-budget settings	Needs reference-derived concept definitions
Distillation inference	Unauthorized distillation attacks	Nothing; trains shadow models for calibration	TPR 1.0 at α=0.02 for a 7B teacher vs 3B suspects	Shadow-model training is the real cost
UTT	KD bound to one architecture and checkpoint	No parameters; a natural-language Primer	KernelBench 9.4%→48.6%; Omni-MATH-2 27.6%→51.7%	Requires paired teacher-student evals to extract gaps
Dense-to-MoE	Inference cost per token	MLP layers reshaped into experts	~40-44% MLP compute; ppl 14.66→19.20 on WikiText-2	Quality gap; some layers must stay dense

Notice what is missing from all four: pretraining compute. PASS changes the data you choose. UTT changes the artifact you produce. Distillation inference changes the audit you run. The MoE conversion changes the architecture you serve. That is the pattern. Post-training is where you get more model for less compute, and increasingly, where you figure out whether the model you're looking at is even your own.

Common pitfalls

1. Treating instruction selection as uniform sampling. If concepts follow a long-tail prior barrier distribution, random or difficulty-ranked sampling over-invests in head concepts the model barely struggles with. If your eval shows the same concepts failing after every fine-tuning run, the problem is selection, not the model. Use an evidence-weighted allocation like PASS.

2. Forcing every layer through the MoE conversion. The final layer, and occasionally an early outlier like SmolLM2's layer 3, degrades sharply when converted to experts. Leaving them dense costs a little MLP compute and saves a lot of quality. Use the perplexity delta per layer to decide, not a blanket policy.

3. Expecting a converted MoE to match the dense original. At 40-44% of MLP compute you are trading quality for speed. The perplexity hit in this experiment was 4-6 points, and at the 0.5B scale that is nearly pure loss because the dense model is already fast. The trade only pays off where inference cost dominates, which means larger bases.

4. Reusing distillation-attribution thresholds across teachers. The shadow-model calibration in the study is specific to a 7B teacher and 3B suspects of particular architectures. The p-value conversion is the method; the threshold is not transferable. Retrain shadow models for your teacher-student pair.

5. Treating the Primer as a substitute for a working student. UTT lifts a student from 9.4% to 48.6% on KernelBench, which is remarkable, but it doesn't turn a broken base model into a strong one. The Primer upgrades a student; it does not replace training.

One thing to remember

The thread connecting these four results is that post-training is becoming a measurement discipline. PASS measures which concepts lack evidence. Distillation inference measures how closely a suspect predicts a teacher's reasoning traces. UTT measures the gap between student and teacher, then writes the fix down in natural language. Even the upcycling experiment is a measurement problem: which layers survive sparsity, and how much quality is a 40% FLOP reduction worth. None of these results required pretraining-scale compute, and that is exactly the point.

The Bottom Line

• If you run a fine-tuning pipeline with a fixed instruction budget, replace uniform or difficulty-ranked sampling with an evidence-weighted adaptive policy like PASS. Your tail concepts are where accuracy disappears, and adaptive allocation is the only strategy tested here that consistently beats seven baselines across four settings.
• If you serve a proprietary model behind an API, set up distillation inference as a recurring audit rather than an after-the-fact investigation. Training the shadow models is a one-time cost, and at a 7B teacher scale you catch distilled 3B suspects at a 1.0 true positive rate while holding false positives at 2%.
• If you're a solo developer on a consumer GPU, skip dense-to-MoE below the 1B scale, the perplexity hit cancels most of the compute win, and reach for UTT-style textual distillation instead. Watch for community upcycling of 1-3B models in the coming months. As base size and expert training time grow, that quality gap is the number to track.