Efficient Hypergradient Descent for Inverse Reinforcement Learning
The paper replaces an expensive IRL hypergradient step with a Fisher-based approximation that can be sketched instead of stored outright.
The authors frame inverse reinforcement learning as bilevel optimization, where policy training sits inside reward learning. Their key claim is that, at the inner optimum, the inner Hessian is proportional to the policy’s Fisher information matrix. They then approximate the needed inverse-Fisher-vector product with a streaming spectral sketch to avoid building the full Fisher matrix. In tests across discrete and continuous control tasks, the method matched competitive policy results and produced strong reward rankings while reducing curvature-storage costs. ArXiv · AI/CL/LG's note
The authors frame inverse reinforcement learning as bilevel optimization, where policy training sits inside reward learning. Their key claim is that, at the inner optimum, the inner Hessian is proportional to the policy’s Fisher information matrix. They then approximate the needed inverse-Fisher-vector product with a streaming spectral sketch to avoid building the full Fisher matrix. In tests across discrete and continuous control tasks, the method matched competitive policy results and produced strong reward rankings while reducing curvature-storage costs. ArXiv · AI/CL/LG's note
score 4