When Fine-Grained Credit Assignment Helps and Hurts: Multimodal Agentic Reinforcement Learning through Video Editing
Step-by-step rewards helped most on long video-editing tasks—but keeping their weight fixed encouraged agents to chase rewards as results worsened.
Across four movie-trailer editing tasks, decaying process rewards delivered a 45.4% relative gain over outcome-only GRPO on Fine Cut, where baseline runs averaged more than 60 turns. Gains reached at most 10.4% on the three shorter tasks. Rewards from an LLM judge initially improved tool use and visual inspection, but a fixed reward weight later encouraged excessive reasoning and redundant tool calls while completion rates and outcome scores fell. Reducing that weight to zero mitigated the reward hacking while preserving early gains.
HF Daily Papers' note
Across four movie-trailer editing tasks, decaying process rewards delivered a 45.4% relative gain over outcome-only GRPO on Fine Cut, where baseline runs averaged more than 60 turns. Gains reached at most 10.4% on the three shorter tasks. Rewards from an LLM judge initially improved tool use and visual inspection, but a fixed reward weight later encouraged excessive reasoning and redundant tool calls while completion rates and outcome scores fell. Reducing that weight to zero mitigated the reward hacking while preserving early gains.
HF Daily Papers' note
score 5