Fast Weight Attention for Continual Learning
The paper reframes recurrent memory updates as an online learning rule with stricter timing.
It argues that, for prefix prediction, the fast-memory example at step `t` should pair the previous key with the current value, not the same-step key and value. The authors derive normalized first-order update families for regression and inner-product objectives, named Falcon-1 through Falcon-3 and their “A” variants. They also give recurrent and parallel forms, plus a decay renormalization method for numerical stability. Reported variants stay competitive on language modeling and improve length extrapolation on variable-digit addition. HF Daily Papers' note
It argues that, for prefix prediction, the fast-memory example at step `t` should pair the previous key with the current value, not the same-step key and value. The authors derive normalized first-order update families for regression and inner-product objectives, named Falcon-1 through Falcon-3 and their “A” variants. They also give recurrent and parallel forms, plus a decay renormalization method for numerical stability. Reported variants stay competitive on language modeling and improve length extrapolation on variable-digit addition. HF Daily Papers' note
score 5