Don't Mask the Environment: Observation Supervision Changes How Agents Explore Under RL
ActObs trains on environment observations as targets during SFT, and the paper says that changes what RL can later explore.
The authors compare standard action-only supervision with ActObs, which also predicts observation tokens already present in trajectories. After SFT the methods look similar, but after GRPO ActObs improves pass@k on Terminal-Bench 2.0 for Qwen3-4B and raises Qwen3-8B pass@16 by 3.4 points while solving more distinct tasks. The paper also reports gains on unseen aider-polyglot code-editing tasks at 4B. Its explanation is that joint action-observation supervision preserves consequence prediction, keeps more entropy during RL, and leaves the final policy closer to its SFT start. HF Daily Papers' note
The authors compare standard action-only supervision with ActObs, which also predicts observation tokens already present in trajectories. After SFT the methods look similar, but after GRPO ActObs improves pass@k on Terminal-Bench 2.0 for Qwen3-4B and raises Qwen3-8B pass@16 by 3.4 points while solving more distinct tasks. The paper also reports gains on unseen aider-polyglot code-editing tasks at 4B. Its explanation is that joint action-observation supervision preserves consequence prediction, keeps more entropy during RL, and leaves the final policy closer to its SFT start. HF Daily Papers' note
score 5