Reward Structure Shapes the Interaction Between Episodic Exploration and Neural Memory in Reinforcement Learning
The paper argues that exploration bonuses only help when memory can turn the extra exposure into usable return.
The authors test episodic exploration bonuses against several neural memory architectures in partially observable reinforcement learning tasks. The same bonus has different effects depending on how the reward supervises the memory content: widening architecture gaps, pushing models to a shared ceiling, or doing nothing. They also find that dense rewards only neutralize a bonus when they directly supervise the latent memory the task needs. A small avoidable penalty on exploratory actions can trap policies in suboptimal stationary behavior, and the bonus resolves it. ArXiv · AI/CL/LG's note
The authors test episodic exploration bonuses against several neural memory architectures in partially observable reinforcement learning tasks. The same bonus has different effects depending on how the reward supervises the memory content: widening architecture gaps, pushing models to a shared ceiling, or doing nothing. They also find that dense rewards only neutralize a bonus when they directly supervise the latent memory the task needs. A small avoidable penalty on exploratory actions can trap policies in suboptimal stationary behavior, and the bonus resolves it. ArXiv · AI/CL/LG's note
score 4