DAMI: Dual-Axis Memory Initialization for Enhanced Long-Context Retrieval in Gated Linear Attention
DAMI improved long-context retrieval accuracy by 5.29–16.71 percentage points without adding trainable parameters.
Across five model families, the gains came against corresponding random-initialization baselines on RULER-style retrieval tests. The method configures memory timescales across attention heads and input sensitivity across layers before training. Controlled Gated DeltaNet experiments found that two parameters governing memory decay remained strongly correlated with their initial values after training. Language modeling, commonsense reasoning and 2K real-world retrieval performance stayed broadly comparable to the baselines.
HF Daily Papers' note
Across five model families, the gains came against corresponding random-initialization baselines on RULER-style retrieval tests. The method configures memory timescales across attention heads and input sensitivity across layers before training. Controlled Gated DeltaNet experiments found that two parameters governing memory decay remained strongly correlated with their initial values after training. Language modeling, commonsense reasoning and 2K real-world retrieval performance stayed broadly comparable to the baselines.
HF Daily Papers' note
score 4