Continual Learning Mechanisms Compose for Long-Horizon Memorization
Combining several anti-forgetting methods beat any one of them on 100-task memory retention.
The paper defines “long-horizon memorization” as learning 100 query-answer tasks through continual fine-tuning, without old examples or task IDs at inference. Naive sequential fine-tuning ended at 1.2% average final retention. The best setup combined data, function, and weight anchors with merged LoRA, reaching 34.9% retention. The authors report the data anchor and merged LoRA as the largest contributors, with super-additive interaction across all three datasets. HF Daily Papers' note
The paper defines “long-horizon memorization” as learning 100 query-answer tasks through continual fine-tuning, without old examples or task IDs at inference. Naive sequential fine-tuning ended at 1.2% average final retention. The best setup combined data, function, and weight anchors with merged LoRA, reaching 34.9% retention. The authors report the data anchor and merged LoRA as the largest contributors, with super-additive interaction across all three datasets. HF Daily Papers' note
score 4