A Tropical Geometry View of Forgetting: A Per-Unit Projector for Knowledge-Preserving Fine-Tuning
The paper argues that forgetting can be reduced by constraining fine-tuning updates per ReLU unit, rather than with one shared layer-wide subspace.
Its projector protects only the old tokens a given unit actually fires on, which the authors say is cheaper than paying for the union of tokens in every row. On OPT-1.3b, it forgets less than Adam-NSCL across six matched budgets, with the reported gap growing from 1.1x to 4.3x. With far fewer constrained directions, it also halves Adam-NSCL’s forgetting at GPM’s energy threshold. The authors say the open/closed token partition is the key variable, and their tests find open-token sets beating random and gate-blind alternatives across all seed pairs. ArXiv · AI/CL/LG's note
Its projector protects only the old tokens a given unit actually fires on, which the authors say is cheaper than paying for the union of tokens in every row. On OPT-1.3b, it forgets less than Adam-NSCL across six matched budgets, with the reported gap growing from 1.1x to 4.3x. With far fewer constrained directions, it also halves Adam-NSCL’s forgetting at GPM’s energy threshold. The authors say the open/closed token partition is the key variable, and their tests find open-token sets beating random and gate-blind alternatives across all seed pairs. ArXiv · AI/CL/LG's note
score 4