SpectralShift: Effective Context Window Extension of Gated DeltaNet via Spectral Reparameterization
SpectralShift targets Gated DeltaNet’s decay spectrum so longer dependencies survive without losing fast state clearing.
The paper argues that simply continuing long-context training misses how linear attention states decay over time. Its method reinitializes alpha projections to widen the slow spectral band, then scales their learning rate during continual pretraining. Experiments in the abstract are described as consistently improving long-context capability for GDNs. HF Daily Papers' note
The paper argues that simply continuing long-context training misses how linear attention states decay over time. Its method reinitializes alpha projections to widen the slow spectral band, then scales their learning rate during continual pretraining. Experiments in the abstract are described as consistently improving long-context capability for GDNs. HF Daily Papers' note
score 5