Pretraining Latent Information Feedback Transformers with Teacher Supervision
LIFT trains language models to pass predicted latent state forward, not just the decoded token.
The paper frames standard Transformers as losing useful deep-layer information between generation steps. Its method pairs each token with a dense state derived from an off-the-shelf teacher model’s next-token distribution, then trains the model to predict both the next token and the next state. Because those states are precomputed, pretraining stays parallel across positions. In tests from 135M to 1B parameters, LIFT beats standard Transformers and baselines under a token-matched budget, and is competitive or better under compute matching. Source: ArXiv · AI/CL/LG's note
The paper frames standard Transformers as losing useful deep-layer information between generation steps. Its method pairs each token with a dense state derived from an off-the-shelf teacher model’s next-token distribution, then trains the model to predict both the next token and the next state. Because those states are precomputed, pretraining stays parallel across positions. In tests from 135M to 1B parameters, LIFT beats standard Transformers and baselines under a token-matched budget, and is competitive or better under compute matching. Source: ArXiv · AI/CL/LG's note
score 5