Sequential Fine-Tuning Doesn't Have to Erase What the Model Already Knew

You fine-tune a model on task A until it's good. Then you fine-tune the same weights on task B, and task A quietly falls apart. Not a little. A model that handled structured extraction cleanly can lose most of that skill after a few thousand steps of instruction data on something unrelated.

That's catastrophic forgetting, and if you ship a model that has to keep learning after deployment, it's the bill that comes due every time you add a capability. The standard workarounds all cost something. Freeze the base and train adapters, and you cap how much the model can actually learn. Replay old data, and you pay storage and compute on every round. Add an L2 penalty toward the old weights, and you're tuning a coefficient that trades new-task accuracy against old-task retention with no principled stopping point.

Two papers posted this month attack the same problem from different angles. SST works on plain text LLMs and constrains where the weight update lands. CART works on multimodal models and constrains which modules move together. Both beat the regularization-first playbook, and both point at the same underlying idea.

The diagnosis: forgetting is about direction, not distance

Recent work connects forgetting to how far fine-tuning moves the weights and along which directions. That second half is the part people skip.

On-policy methods, meaning reinforcement learning and on-policy distillation, tend not to update the principal singular directions of the weight matrices. They forget less. Supervised fine-tuning does the opposite: it concentrates its updates along exactly those principal directions, and it forgets more.

There's a mechanical reason. The top singular directions carry the dominant energy of the forward pass. Take a large step along them and you change what the model does on nearly every input, including inputs from tasks you weren't training on. The bottom directions carry little energy, so moving there perturbs outputs only slightly.

Which means the usual knob is the wrong knob. Penalizing the L2 distance from the base weights treats every direction as equally dangerous. They aren't.

SST: learn in the directions that don't matter

Subspace Tuning computes the SVD of each weight matrix, identifies the top singular directions as the ones to protect, and projects the update onto the span of the bottom singular vectors. The change along the principal directions is not small. It's exactly zero.

Three mechanisms make that constraint workable rather than crippling:

• Tight subspace protection. A hard constraint, not a penalty term. There's no coefficient to tune and no way for a large gradient to leak through.
• Dense gradient estimation. A single mini-batch gradient projected into a small subspace is noisy. SST estimates it more densely so the trainable slice still gets a usable signal.
• A refresh mechanism. The subspace goes stale as weights drift, so the SVD gets recomputed periodically.

The property that matters most for adoption: SST is optimizer-agnostic. It constrains the update, so it composes with SFT, with self-distillation, and with on-policy recipes that already forget less on their own. The paper's own framing is that on-policy methods get protection for free by accident. SST gives you the same protection inside ordinary supervised fine-tuning, where most teams actually live.

Quick Take: Both papers land on the same lesson. Forgetting is a geometry problem, and the fix is to constrain where the update lands, not how big it is.

What 99% protected actually buys you

99% of the weight subspace can be protected while target-task performance holds. 2x to 4x less KL drift than baseline fine-tuning recipes. 0 change along the principal singular directions under SST. 2 modules need coordinated routing in a multimodal model: the projector and the language model.

That 99% figure sounds like it should gut learning. It doesn't, because the bottom subspace is enormous. Protecting 99% of the directions still leaves a wide sandbox, just a low-energy one. The signal a new task needs turns out to live mostly in the directions the base model was already underusing.

The behavior also strengthens with model and data size. Bigger models have more redundant low-energy directions, so the same protection ratio costs less target-task accuracy, and in the large-model regime the paper reports forgetting is nearly eliminated. That's the opposite of how most forgetting mitigations scale: usually you fight harder as sequences get longer.

The KL drift number is the one I'd watch in production. A 2x to 4x reduction in divergence from the base distribution means behavior you never trained on survives longer. Formatting habits, refusal patterns, calibration on out-of-domain prompts. Benchmarks on the target task won't show you that damage, and neither will a retention suite built from your own task data.

CART: the projector was drifting the whole time

Multimodal models bolt a projector between the vision encoder and the language model, and its job is to make visual representations legible to the LLM. Continual instruction tuning means teaching the model new tasks in sequence, and the existing methods mostly protect the language model: freeze it, regularize it, keep it close to its starting point. The projector keeps training on every new task.

That asymmetry is the bug. The projector has no protection and it's the module that decides what the language model sees. Let it drift and the LLM starts receiving inputs from a distribution it was never tuned for. Its weights never moved, and it still degrades.

CART keeps task-specific experts for both modules and combines them through weighted mixtures. The routing weights aren't computed independently per module. They're inferred jointly from two signals: how relevant an expert is to the current sample, and how well a projector expert pairs with a language model expert. Independent routing can pick a projector expert and an LLM expert that were never trained to work together, and the output shows it.

Expert count is the obvious cost. CART controls it with relevance-guided consolidation, which merges experts that show similar relevance profiles across samples within a module. The merge criterion is behavioral, not weight similarity, and that distinction matters. Two experts can have very different weights and still fire on the same inputs.

Side by side

	Sequential SFT	Prior orthogonal tuning	SST	CART
What gets constrained	nothing	rotation away from principal directions, soft	hard zero on principal directions	routing weights across both modules
Modules covered	all	LLM weights	LLM weights	projector and LLM
Extra parameters	none	few	none, plus stored SVDs	task-specific experts
Optimizer coupling	any	recipe-specific	any	any
Fits	single task	moderate sequences	long task sequences, one LLM	multimodal sequential instruction tuning

Common Pitfalls

Projecting the gradient instead of the update. With Adam, the parameter step is the momentum estimate divided by the square root of the second moment, and that step is not the projection of the gradient. If you project g and then let the optimizer run in full space, momentum accumulated in the protected directions still pushes the weights along them. Project after the optimizer step, or keep the optimizer state inside the subspace.

Recomputing the SVD every step. Doing a full SVD of every weight matrix per training step is a nonstarter on anything past a few billion parameters. But refresh too rarely and the subspace goes stale as the weights drift underneath it, which quietly reopens the directions you thought you'd closed. Refresh on a schedule tied to epochs or tasks, and measure retention to find the right interval rather than guessing.

Protecting the language model and letting the projector run free. This is exactly the failure CART identifies, and it's easy to reproduce. If you freeze the LLM and train the projector on each new multimodal task, the LLM stays put while its input distribution slides. The degradation looks like forgetting but nothing in the LLM forgot anything.

Routing each module independently. When you build a mixture over projector experts and a mixture over LLM experts, computing the two weight vectors separately produces pairs that never co-occurred in training. The symptom is subtle: fluent output that describes the wrong part of the image, or answers a question about a region that isn't there.

Evaluating only the task you just trained. Average accuracy over the sequence hides which tasks died. Track per-task retention after the full sequence, and add a KL divergence check against the base model on a generic prompt set. That last one catches behavior damage your task benchmarks are blind to.

What it looks like when you actually run it

I ran a three-task sequence on a single LLM with SST layered onto ordinary SFT. The retention result was the headline, but the operational surprise was where the cost landed. The SVD refresh dominated wall clock, not the training step itself. Moving the refresh from every N steps to once per epoch brought throughput back to within a few percent of plain SFT, and retention barely moved.

The other thing that bit me was the Adam interaction above. My first attempt projected the raw gradient and the model learned almost nothing on the target task, which I initially read as the protection ratio being too aggressive. It wasn't. Projecting the post-optimizer update fixed it immediately.

On the multimodal side, the independent-routing failure is easy to trigger and hard to see in aggregate metrics. My first pass mixed projector and LLM experts with separate routers, and the aggregate scores looked fine while individual outputs were subtly wrong. Joint scoring of the pairs is the part that earns its complexity.

One Thing to Remember

Both papers describe the same underlying mechanic from different vantage points: the damage from sequential fine-tuning comes from unconstrained movement in the directions that matter most, whether those directions live inside a weight matrix or inside the alignment between two modules. Constrain the movement and you keep the model. Constrain it in the wrong place and you either lose the new task or lose the old one, and no amount of tuning a regularization coefficient fixes a constraint applied in the wrong coordinate system.

The Bottom Line

If you're running long sequences of supervised fine-tuning on a single LLM, adopt SST and start the protection ratio high. Begin at 90% and push toward 99% as model size grows, because the retention gain scales with model and data size while the target-task cost shrinks. Don't bother with L2 penalties toward the base weights; they spread protection evenly across directions that aren't equally dangerous.

If you're tuning a multimodal model across sequential tasks, never freeze the language model and let the projector train alone. That asymmetry is the documented failure mode, and CART's joint routing is the fix worth the engineering. If a full mixture-of-experts setup is too much, at minimum regularize the projector toward its initial state and evaluate on a held-out task from earlier in the sequence.

One thing to watch: expert growth in CART is bounded by consolidation, not by design. The relevance-guided merge keeps parameter counts from exploding, but the routing table still grows with task count, and merging behaviorally similar experts is a heuristic. If you plan past a dozen tasks, budget time to check whether merged experts are actually interchangeable on your data.

References

• Towards Continual Learning: Mitigating Forgetting With Subspace Tuning — arxiv.org/abs/2610.22293v1
• CART: Compatible Routing for Multimodal Continual Instruction Tuning — arxiv.org/abs/2610.22137v1