ARM: Attention with Routed-Memory for Learnable Sparse Control
ARM replaces hard KV-cache eviction with learned, fixed-size routed memory.
The paper proposes a differentiable memory hierarchy that uses Gumbel-Softmax routing to choose memory slots and sigmoid-gated updates to blend new and stored information. It also trains an inference-time policy to vary how much memory the model accesses depending on the input. The authors report better performance and efficiency than fixed KV-caching baselines on commonsense and long-context reasoning benchmarks, with gains in memory use and generation latency. Accepted to ICML 2026. ArXiv · AI/CL/LG's note
The paper proposes a differentiable memory hierarchy that uses Gumbel-Softmax routing to choose memory slots and sigmoid-gated updates to blend new and stored information. It also trains an inference-time policy to vary how much memory the model accesses depending on the input. The authors report better performance and efficiency than fixed KV-caching baselines on commonsense and long-context reasoning benchmarks, with gains in memory use and generation latency. Accepted to ICML 2026. ArXiv · AI/CL/LG's note
score 5