TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agents
TRACE assigns reward at each tool-call boundary, instead of judging a long agent run only by its final answer.
The method uses a frozen reference model to estimate state values from gold-answer log probabilities, then turns changes in those values into per-action rewards. The authors say this avoids adding a critic or training process labels. On BrowseComp-Plus, it lifts Qwen3-4B from 7.2 to 35.6 and Qwen3-30B-A3B from 8.4 to 42.6. They report the learned search behavior also transfers to open-web benchmarks, with earlier gains and faster RL convergence. ArXiv · AI/CL/LG's note
The method uses a frozen reference model to estimate state values from gold-answer log probabilities, then turns changes in those values into per-action rewards. The authors say this avoids adding a critic or training process labels. On BrowseComp-Plus, it lifts Qwen3-4B from 7.2 to 35.6 and Qwen3-30B-A3B from 8.4 to 42.6. They report the learned search behavior also transfers to open-web benchmarks, with earlier gains and faster RL convergence. ArXiv · AI/CL/LG's note
score 5