SFT Conflicts, RL Coexists: A Theoretical and Empirical Analysis of Multi-Task Learning for LLMs
The paper argues RL keeps multi-task LLM training from colliding because its task updates stay sparse and nearly orthogonal.
The authors contrast that with SFT, which they say shows severe task conflicts in multi-stage training. Their explanation is both empirical and theoretical: SFT interference scales with gradient magnitude, while RL interference is bounded by variance from advantage normalization and on-policy optimization. They propose Parallel-RL as a way to decouple multi-task training for better efficiency and flexibility. HF Daily Papers' note
The authors contrast that with SFT, which they say shows severe task conflicts in multi-stage training. Their explanation is both empirical and theoretical: SFT interference scales with gradient magnitude, while RL interference is bounded by variance from advantage normalization and on-policy optimization. They propose Parallel-RL as a way to decouple multi-task training for better efficiency and flexibility. HF Daily Papers' note
score 5