Towards Robust Reinforcement Learning for Small-Scale Language Model Agents
The paper pins small-model RL failures on pipeline bugs, precision overflow, and reward-model collapse, then reports stable PPO runs after targeted fixes.
The authors trained 15 model-and-corpus setups across Pythia and SmolLM2 models from 70M to 410M parameters. They identify three reproducible failure modes: frozen LoRA parameters, bfloat16 importance-ratio overflow, and policy collapse from reward-model error. Their fixes include adapter merge-and-reinitialization, float32 PPO updates, reward whitening, ratio guards, and rollback. They argue PPO success at this scale depends on a fluent supervised prior and a useful reward signal, not parameter count alone. HF Daily Papers' note
The authors trained 15 model-and-corpus setups across Pythia and SmolLM2 models from 70M to 410M parameters. They identify three reproducible failure modes: frozen LoRA parameters, bfloat16 importance-ratio overflow, and policy collapse from reward-model error. Their fixes include adapter merge-and-reinitialization, float32 PPO updates, reward whitening, ratio guards, and rollback. They argue PPO success at this scale depends on a fluent supervised prior and a useful reward signal, not parameter count alone. HF Daily Papers' note
score 4