Depth Memory
The paper recasts transformer depth as a memory problem, then proposes Linear Depth Memory as the new retrieval mechanism.
Depth Memory is presented as a framework covering PreLN transformers, mHC, and Attention Residuals through streams that carry representations across layers. Linear Depth Memory adds a residual stream, a linear associative memory stream, and an inverse covariance stream, with retrieval framed as recursive least squares. The authors also describe a hardware-aware implementation meant to cut peak activation memory and memory traffic. In their reported 400M and 1.5B-scale tests, LDM gets lower perplexity than the listed baselines while keeping competitive zero-shot accuracy. ArXiv · AI/CL/LG's note
Depth Memory is presented as a framework covering PreLN transformers, mHC, and Attention Residuals through streams that carry representations across layers. Linear Depth Memory adds a residual stream, a linear associative memory stream, and an inverse covariance stream, with retrieval framed as recursive least squares. The authors also describe a hardware-aware implementation meant to cut peak activation memory and memory traffic. In their reported 400M and 1.5B-scale tests, LDM gets lower perplexity than the listed baselines while keeping competitive zero-shot accuracy. ArXiv · AI/CL/LG's note
score 5