Memory Decoder at Scale: A Pretrained, Parametric Long-Term Memory
A scaled memory module beat much larger base-model scaling in the paper’s tests.
The authors scale Memory Decoder up to 6.9B parameters and pretrain it on 300B tokens. They say standard Faiss indexing and search became infeasible at that scale, so they built a distributed indexing and retrieval pipeline with sparse batch-wise loading of kNN distributions. Paired with Pythia-410M, a 6.9B general memory raised the average score from 29.86 to 37.34 across 17 benchmarks, narrowly above Pythia-12B while using fewer total parameters. HF Daily Papers' note
The authors scale Memory Decoder up to 6.9B parameters and pretrain it on 300B tokens. They say standard Faiss indexing and search became infeasible at that scale, so they built a distributed indexing and retrieval pipeline with sparse batch-wise loading of kNN distributions. Paired with Pythia-410M, a 6.9B general memory raised the average score from 29.86 to 37.34 across 17 benchmarks, narrowly above Pythia-12B while using fewer total parameters. HF Daily Papers' note
score 5