Latent Lattice Language Models
A 406M-parameter model cut validation perplexity from 15.41 to 14.40 by adding computational states that persist across tokens.
The “Lattice of Latents” adds recurrent streams at each layer that directly influence attention, KV-cache writes, MLP computation and decoding. The comparison used a parameter-matched Transformer trained on 20 billion FineWeb-Edu tokens. Combined with hyper-connections and input residuals, the Lattice reached 13.61 perplexity, versus 14.22 with those enhancements alone. It also improved every tested native state size across four recurrent and hybrid architectures.
ArXiv · AI/CL/LG's note
The “Lattice of Latents” adds recurrent streams at each layer that directly influence attention, KV-cache writes, MLP computation and decoding. The comparison used a parameter-matched Transformer trained on 20 billion FineWeb-Edu tokens. Combined with hyper-connections and input residuals, the Lattice reached 13.61 perplexity, versus 14.22 with those enhancements alone. It also improved every tested native state size across four recurrent and hybrid architectures.
ArXiv · AI/CL/LG's note
score 5