Failure-Transparent Agents: Benchmarking Post-Failure Reporting in Tool-Using Language Models
A structured evidence contract cut false success reports to 0.8% in the benchmark.
The paper introduces FTA, a controlled test for whether tool-using agents admit failure after a required tool has failed. It fixes the failed observation and required evidence state before generation, so unsupported claims can be audited directly. Across 3,600 human-annotated responses, baseline false-success rates were 22.8%, versus 9.3% with a transparency instruction and 0.8% with the evidence contract. Useful responses also rose under the stricter policy. ArXiv · AI/CL/LG's note
The paper introduces FTA, a controlled test for whether tool-using agents admit failure after a required tool has failed. It fixes the failed observation and required evidence state before generation, so unsupported claims can be audited directly. Across 3,600 human-annotated responses, baseline false-success rates were 22.8%, versus 9.3% with a transparency instruction and 0.8% with the evidence contract. Useful responses also rose under the stricter policy. ArXiv · AI/CL/LG's note
score 5