Continuum Attention
Integrating over a continuous memory curve lowered validation loss against both softmax and focal attention in the reported tests.
Continuum Attention replaces discrete key and value lookup with integration over a piecewise linear curve, computing normalized attention exactly using closed-form expressions. Mean validation negative log likelihood fell by 0.0200 nats per byte against softmax and 0.0480 against focal attention, winning all paired comparisons. The method retains the standard Transformer decoder and quadratic sequence complexity. Whether the gains hold at larger scales remains open.
HF Daily Papers' note
Continuum Attention replaces discrete key and value lookup with integration over a piecewise linear curve, computing normalized attention exactly using closed-form expressions. Mean validation negative log likelihood fell by 0.0200 nats per byte against softmax and 0.0480 against focal attention, winning all paired comparisons. The method retains the standard Transformer decoder and quadratic sequence complexity. Whether the gains hold at larger scales remains open.
HF Daily Papers' note
score 5