CORD: Conflict-Aware Optimization for Joint RL and Distillation in Agents
CORD raised ALFWorld success by 1.5–15.9 percentage points across three joint training recipes.
Reinforcement learning and self-distillation can produce gradients that partly cancel, costing local optimization progress. CORD selectively corrects conflicting components, separating which objective is protected from when correction is triggered. On WebShop, it improved two of three tested pairings; SDAR success rose from 60.7% to 64.6%. The authors report that the distillation objective and its mixing weight still matter after correction.
ArXiv · AI/CL/LG's note
Reinforcement learning and self-distillation can produce gradients that partly cancel, costing local optimization progress. CORD selectively corrects conflicting components, separating which objective is protected from when correction is triggered. On WebShop, it improved two of three tested pairings; SDAR success rose from 60.7% to 64.6%. The authors report that the distillation objective and its mixing weight still matter after correction.
ArXiv · AI/CL/LG's note
score 4