Towards Continual Learning: Mitigating Forgetting With Subspace Tuning
Protecting up to 99% of the weight subspace improved retention while preserving strong performance on new tasks, the authors report.
Subspace Tuning restricts weight updates to the bottom singular subspace, leaving principal singular directions unchanged. It works with any optimizer and can be combined with supervised fine-tuning and self-distillation. The authors report that retention improves with model and data size, with forgetting nearly eliminated in the large-model regime. They also report 2–4 times less KL divergence from the base model’s distribution than baseline recipes.
ArXiv · AI/CL/LG's note
Subspace Tuning restricts weight updates to the bottom singular subspace, leaving principal singular directions unchanged. It works with any optimizer and can be combined with supervised fine-tuning and self-distillation. The authors report that retention improves with model and data size, with forgetting nearly eliminated in the large-model regime. They also report 2–4 times less KL divergence from the base model’s distribution than baseline recipes.
ArXiv · AI/CL/LG's note
score 5