Sample complexity of variance-reduced policy gradient: weaker assumptions and lower bounds
Defensive importance sampling matches the faster variance-reduced REINFORCE rate without assuming bounded variance for ordinary importance weights.
The paper introduces Defensive Policy Gradient, which reaches \(O(\epsilon^{-3})\) sample complexity for finding an \(\epsilon\)-stationary point. It contrasts that with vanilla REINFORCE’s \(O(\epsilon^{-4})\) rate under the oracle conditions considered. The authors also give black-box lower bounds showing when those two rates are unavoidable in their generalized model. They note the lower bounds do not directly cover the classical MDP interaction setting, but argue they still support the separation at the oracle level. ArXiv · AI/CL/LG's note
The paper introduces Defensive Policy Gradient, which reaches \(O(\epsilon^{-3})\) sample complexity for finding an \(\epsilon\)-stationary point. It contrasts that with vanilla REINFORCE’s \(O(\epsilon^{-4})\) rate under the oracle conditions considered. The authors also give black-box lower bounds showing when those two rates are unavoidable in their generalized model. They note the lower bounds do not directly cover the classical MDP interaction setting, but argue they still support the separation at the oracle level. ArXiv · AI/CL/LG's note
score 4