Kalman Delta Networks: Uncertainty-aware Associative Memory
The paper recasts linear-attention memory updates as Kalman filtering so each write can depend on tracked uncertainty.
Kalman Delta Networks propagate both the associative memory state and its uncertainty, using a Kalman gain to scale residual writes by accumulated evidence and observation reliability. The authors say standard Delta-style updates fall out as a special case that skips covariance tracking. Because exact covariance updates are too dense for GPU-friendly scans, they introduce diagonal and isotropic approximations with scan-compatible uncertainty recurrences. In controlled pretraining at 750M and 1.3B parameters, the KDN variants improve perplexity and mean downstream accuracy over state-of-the-art linear-attention models. HF Daily Papers' note
Kalman Delta Networks propagate both the associative memory state and its uncertainty, using a Kalman gain to scale residual writes by accumulated evidence and observation reliability. The authors say standard Delta-style updates fall out as a special case that skips covariance tracking. Because exact covariance updates are too dense for GPU-friendly scans, they introduce diagonal and isotropic approximations with scan-compatible uncertainty recurrences. In controlled pretraining at 750M and 1.3B parameters, the KDN variants improve perplexity and mean downstream accuracy over state-of-the-art linear-attention models. HF Daily Papers' note
score 5