Video-RSI: Recursive Self-Improvement of Video Understanding Agents via Harness Evolution
Video-RSI lets a video agent revise the harness that decides what evidence it gathers.
The paper says current execution traces can hide why a video-understanding agent failed, because they only show what the existing harness chose to observe. Video-RSI has the model revisit training videos, test competing failure explanations, and turn the findings into reusable harness changes. The retention step weighs both answer accuracy and visual cost. In the reported benchmark settings, the evolved agent improved accuracy while processing fewer frames. HF Daily Papers' note
The paper says current execution traces can hide why a video-understanding agent failed, because they only show what the existing harness chose to observe. Video-RSI has the model revisit training videos, test competing failure explanations, and turn the findings into reusable harness changes. The retention step weighs both answer accuracy and visual cost. In the reported benchmark settings, the evolved agent improved accuracy while processing fewer frames. HF Daily Papers' note
score 4