First Passage Attention
A 110M-parameter model achieved perfect passkey retrieval at eight times its training length, with its key–value cache compressed to about 25%.
First Passage Attention is designed to erase intermediate context while preserving older, structurally relevant tokens. It penalises attention scores using the deepest dip in a learned scalar field across the intervening tokens, rather than accumulating decay along that path. In synthetic tests, one-layer FPA heads learned the target token sets while comparable softmax and Forgetting Transformer baselines failed on every seed. The language model also matched the Forgetting Transformer in perplexity.
HF Daily Papers' note
First Passage Attention is designed to erase intermediate context while preserving older, structurally relevant tokens. It penalises attention scores using the deepest dip in a learned scalar field across the intervening tokens, rather than accumulating decay along that path. In synthetic tests, one-layer FPA heads learned the target token sets while comparable softmax and Forgetting Transformer baselines failed on every seed. The language model also matched the Forgetting Transformer in perplexity.
HF Daily Papers' note
score 5