Horizon-Scalable Q-Learning
HQL follows recorded actions only when its critic rates them at least as highly as sampled alternatives.
One-step value learning can compound small errors over long tasks, while multi-step learning can introduce bias by following suboptimal actions in the dataset. HQL extends its return target when a recorded action passes a comparison against the best of N actions sampled from a behavior-cloning policy; otherwise, it uses a one-step target. The correction requires no action densities, making it compatible with flow and diffusion policies. The authors report avoiding both methods’ failures on a combination-lock task and outperforming prior horizon-reduction methods on long-horizon OGBench and OCBench tasks.
HF Daily Papers' note
One-step value learning can compound small errors over long tasks, while multi-step learning can introduce bias by following suboptimal actions in the dataset. HQL extends its return target when a recorded action passes a comparison against the best of N actions sampled from a behavior-cloning policy; otherwise, it uses a one-step target. The correction requires no action densities, making it compatible with flow and diffusion policies. The authors report avoiding both methods’ failures on a combination-lock task and outperforming prior horizon-reduction methods on long-horizon OGBench and OCBench tasks.
HF Daily Papers' note
score 5