Do You Really Need to Pretrain Q-Functions for Online RL Fine-Tuning?
Naive Q-function pretraining often fails to beat random initialization for online RL fine-tuning.
The paper argues the pretrained Q-function is learning values for the pretrained policy, not the policy that online fine-tuning eventually reaches. That mismatch can survive even after offline value maximization. The authors propose Initialization via Policy Ensemble, using rollouts from multiple diverse policies to bootstrap Q-learning. On continuous-control benchmarks, they report a 1.26x average fine-tuning gain over naive Q-function pretraining. ArXiv · AI/CL/LG's note
The paper argues the pretrained Q-function is learning values for the pretrained policy, not the policy that online fine-tuning eventually reaches. That mismatch can survive even after offline value maximization. The authors propose Initialization via Policy Ensemble, using rollouts from multiple diverse policies to bootstrap Q-learning. On continuous-control benchmarks, they report a 1.26x average fine-tuning gain over naive Q-function pretraining. ArXiv · AI/CL/LG's note
score 4