Don't Mask the Environment: Observation Supervision Changes How Agents Explore Under RL
ActObs trains agents to predict the observation tokens in their own trajectories, and that changes how later RL explores.
The paper says standard SFT ignores environment observations as loss targets, while ActObs supervises them without adding data, parameters, tokens, or forward passes. After SFT the methods look similar, but after GRPO ActObs beats action-only training on Terminal-Bench 2.0 across evaluated pass@k budgets for Qwen3-4B. For Qwen3-8B, it gives up some pass@1 reliability but improves pass@16 by 3.4 points and solves more distinct tasks. The authors trace the gain to preserved consequence prediction: action-only SFT leaves a residual observation gradient and worsens environment prediction, while joint supervision keeps the final RL policy closer to its SFT start. ArXiv · AI/CL/LG's note
The paper says standard SFT ignores environment observations as loss targets, while ActObs supervises them without adding data, parameters, tokens, or forward passes. After SFT the methods look similar, but after GRPO ActObs beats action-only training on Terminal-Bench 2.0 across evaluated pass@k budgets for Qwen3-4B. For Qwen3-8B, it gives up some pass@1 reliability but improves pass@16 by 3.4 points and solves more distinct tasks. The authors trace the gain to preserved consequence prediction: action-only SFT leaves a residual observation gradient and worsens environment prediction, while joint supervision keeps the final RL policy closer to its SFT start. ArXiv · AI/CL/LG's note
score 5