DeltaTTG: Training Multi-Turn Agents with a Token-to-Go Progress Meter
DeltaTTG turns “tokens left until completion” into a reusable progress signal for multi-turn agent training.
The paper says sparse terminal rewards make it hard to know which intermediate steps helped an agent finish a task. DeltaTTG trains a frozen progress meter from recorded trajectories, then uses changes in its Token-to-Go estimates to redistribute outcome-derived advantages across steps. On ALFWorld, PaperSearchQA, and BFCLv3, it beats GRPO by 5.7 points with Qwen3-4B and 7.2 points with Llama-3.1-8B-Instruct, while keeping GPU-hours close to GRPO. The authors also report better sample efficiency, out-of-domain performance, and test-time action selection guidance. HF Daily Papers' note
The paper says sparse terminal rewards make it hard to know which intermediate steps helped an agent finish a task. DeltaTTG trains a frozen progress meter from recorded trajectories, then uses changes in its Token-to-Go estimates to redistribute outcome-derived advantages across steps. On ALFWorld, PaperSearchQA, and BFCLv3, it beats GRPO by 5.7 points with Qwen3-4B and 7.2 points with Llama-3.1-8B-Instruct, while keeping GPU-hours close to GRPO. The authors also report better sample efficiency, out-of-domain performance, and test-time action selection guidance. HF Daily Papers' note
score 5