The Head Complexity of Boolean Functions in Single-Layer Attention
Single-layer attention needs one head per parity bit, even with unlimited dimension and precision.
The paper proves that `k` attention heads can compute `k`-bit parity, but not `(k+1)`-bit parity. Its lower bound does not rely on limiting embedding size or numerical precision. The same obstruction is used for related lower bounds, including multi-hop induction-head tasks. It also gives near-matching bounds for general binary functions: `2^n` heads suffice, while almost all require `Ω(2^n/n^2)`. ArXiv · AI/CL/LG's note
The paper proves that `k` attention heads can compute `k`-bit parity, but not `(k+1)`-bit parity. Its lower bound does not rely on limiting embedding size or numerical precision. The same obstruction is used for related lower bounds, including multi-hop induction-head tasks. It also gives near-matching bounds for general binary functions: `2^n` heads suffice, while almost all require `Ω(2^n/n^2)`. ArXiv · AI/CL/LG's note
score 5