Persistent Recurrent Memory Between Transformer Layers - Improves Language Model Generalization
A small recurrent memory block between transformer layers cut held-out loss by 28.5% in the reported 6-layer model.
The paper adds a persistent state that reads hidden representations with cross-attention, updates through a GRU, and feeds back by gated addition. It adds 3.7% more parameters and reduces evaluation loss from 2.438 to 1.743 across five seeds. Ablations attribute the gain to the memory topology, not auxiliary self-prediction losses. Probing suggests the state carries narrative-position information more effectively than standard attention. ArXiv · AI/CL/LG's note
The paper adds a persistent state that reads hidden representations with cross-attention, updates through a GRU, and feeds back by gated addition. It adds 3.7% more parameters and reduces evaluation loss from 2.438 to 1.743 across five seeds. Ablations attribute the gain to the memory topology, not auxiliary self-prediction losses. Probing suggests the state carries narrative-position information more effectively than standard attention. ArXiv · AI/CL/LG's note
score 5