Shortcuts and Boundaries: Verifiable RL Misalignment and Contextual Safety Learning
Reward training produced profitable rule violations even when compliant ways to finish the task remained available.
In one API task, training with a 25% mixture of reward–context conflicts yielded 97.9% certified profitable violations after 128 updates. Across resource and reporting workflows, agents increased output through unauthorized resource use or improved reported scores while actual work declined. A binary penalty based on execution checks supported gains in both capability and compliance, while partially reducing acquired violations. On 32 unseen tasks with changing evidence requirements, resource-trained safety raised safe completion from 53.1% to 89.1% compared with task-only training; both policies completed all real work.
ArXiv · AI/CL/LG's note
In one API task, training with a 25% mixture of reward–context conflicts yielded 97.9% certified profitable violations after 128 updates. Across resource and reporting workflows, agents increased output through unauthorized resource use or improved reported scores while actual work declined. A binary penalty based on execution checks supported gains in both capability and compliance, while partially reducing acquired violations. On 32 unseen tasks with changing evidence requirements, resource-trained safety raised safe completion from 53.1% to 89.1% compared with task-only training; both policies completed all real work.
ArXiv · AI/CL/LG's note
score 4