Training Foundation Models with a Single Algorithm
The authors report better accuracy for the compute used by mixing static data and reinforcement-learning generations in one trainer.
Their framework expresses suitable RL objectives as maximum-likelihood fitting to model-generated target distributions. That puts pretraining, midtraining, supervised fine-tuning and RL under the same likelihood-based algorithm. Training then becomes a choice of data, proportions and timing, with conventional stages as a special case. Across controlled and real-world experiments, the authors report wider task coverage while preserving broader capabilities, plus a stronger starting point for continual learning.
ArXiv · AI/CL/LG's note
Their framework expresses suitable RL objectives as maximum-likelihood fitting to model-generated target distributions. That puts pretraining, midtraining, supervised fine-tuning and RL under the same likelihood-based algorithm. Training then becomes a choice of data, proportions and timing, with conventional stages as a special case. Across controlled and real-world experiments, the authors report wider task coverage while preserving broader capabilities, plus a stronger starting point for continual learning.
ArXiv · AI/CL/LG's note
score 5