Asking Smarter Questions: Targeted Active Preference Learning for Underspecified Rewards
Targeted preference questions improved reward learning by up to 30.8% over a standard baseline in simulations with few queries.
The method uses teaching data to identify task features a human has poorly specified, then directs preference questions toward those gaps. Each question includes an explanation pointing out which features the human should consider. In a user study, targeted queries and explanations each significantly improved recovery of the intended reward function, with stronger results when combined. The paper was accepted to CoRL 2026.
ArXiv · AI/CL/LG's note
The method uses teaching data to identify task features a human has poorly specified, then directs preference questions toward those gaps. Each question includes an explanation pointing out which features the human should consider. In a user study, targeted queries and explanations each significantly improved recovery of the intended reward function, with stronger results when combined. The paper was accepted to CoRL 2026.
ArXiv · AI/CL/LG's note
score 4