Abstention and Noise Filtering: Two Missing Primitives of Softmax Attention
The paper separates attention gating into abstention and value-side noise filtering, then finds their gains shift with model scale.
Wang tests matched language models from 10M to 350M parameters. Abstention, implemented with a learned per-head sink logit, explains nearly all the gating gain at 10M but fades as models grow. Noise filtering, implemented as a gate on each value, becomes the larger source of benefit at 350M. Models with both primitives perform best at every scale, with negligible added parameters and compatibility with the key-value cache. ArXiv · AI/CL/LG's note
Wang tests matched language models from 10M to 350M parameters. Abstention, implemented with a learned per-head sink logit, explains nearly all the gating gain at 10M but fades as models grow. Noise filtering, implemented as a gate on each value, becomes the larger source of benefit at 350M. Models with both primitives perform best at every scale, with negligible added parameters and compatibility with the key-value cache. ArXiv · AI/CL/LG's note
score 6