Complex KDA: Understanding and Enhancing the Expressivity of Kimi Delta Attention
CKDA gets Kimi Delta Attention to model rotations without paying for a second delta-rule transition.
The paper extends KDA by allowing gates in `[-1,1]` and the delta-rule coefficient beta in `[0,2]`. With those ranges, its channel-wise gate supplies a reflection that lets one delta-rule update realize 2D rotations while staying diagonal-plus-rank-one and non-expansive. The authors prove CKDA can exactly express every orthogonal diagonal-plus-rank-one matrix and can track finite groups isomorphic to subgroups of `SO(3)`. In tests, the combined extension gives the best length extrapolation among KDA range settings on `S_3`, `S_4`, and periodic audio continuation; in language modeling it beats Transformers and other linear RNNs while staying close to the KDA baseline.
HF Daily Papers' note
The paper extends KDA by allowing gates in `[-1,1]` and the delta-rule coefficient beta in `[0,2]`. With those ranges, its channel-wise gate supplies a reflection that lets one delta-rule update realize 2D rotations while staying diagonal-plus-rank-one and non-expansive. The authors prove CKDA can exactly express every orthogonal diagonal-plus-rank-one matrix and can track finite groups isomorphic to subgroups of `SO(3)`. In tests, the combined extension gives the best length extrapolation among KDA range settings on `S_3`, `S_4`, and periodic audio continuation; in language modeling it beats Transformers and other linear RNNs while staying close to the KDA baseline.
HF Daily Papers' note
score 4