Safe Reinforcement Learning for Multi-Turn Clinical Agents: When Dense Rewards Beat Outcome Rewards
Clinical agents could earn higher rewards while asking too few questions and failing on rare, high-risk cases.
The study tested agents in three controlled environments: medication refill triage, diagnostic work-up and dosing titration. In diagnostics, rewarding questions that reduced uncertainty improved held-out value across three training seeds and avoided the high-risk collapse seen with outcome-only training. Separating exploration from decision-making further improved high-risk support at its best checkpoint, but led to excessive questioning in low-acuity cases. The authors found that the right amount of process reward depended on the task, and reinforcement learning did not reliably fix rare behavior gaps inherited from biased supervised data.
ArXiv · AI/CL/LG's note
The study tested agents in three controlled environments: medication refill triage, diagnostic work-up and dosing titration. In diagnostics, rewarding questions that reduced uncertainty improved held-out value across three training seeds and avoided the high-risk collapse seen with outcome-only training. Separating exploration from decision-making further improved high-risk support at its best checkpoint, but led to excessive questioning in low-acuity cases. The authors found that the right amount of process reward depended on the task, and reinforcement learning did not reliably fix rare behavior gaps inherited from biased supervised data.
ArXiv · AI/CL/LG's note
score 4