SleuthVLM: Do We Really Need Audio-in-Video for Omni Video Understanding?
SleuthVLM reports 75.0% average accuracy across four video benchmarks without feeding audio directly to its vision-language model.
The system transcribes speech into text and lets the model investigate the video without a predefined workflow. On benchmarks covering videos up to 90 minutes, it finished 2.6 percentage points above Gemini 3.8 Flash and scored 77.8% on Video-MME-v2. A version using an open-source 27B vision-language model also beat all open-source omni models in the reported comparison. The authors argue that better agentic ability, rather than direct audio access alone, is the key bottleneck.
HF Daily Papers' note
The system transcribes speech into text and lets the model investigate the video without a predefined workflow. On benchmarks covering videos up to 90 minutes, it finished 2.6 percentage points above Gemini 3.8 Flash and scored 77.8% on Video-MME-v2. A version using an open-source 27B vision-language model also beat all open-source omni models in the reported comparison. The authors argue that better agentic ability, rather than direct audio access alone, is the key bottleneck.
HF Daily Papers' note
score 6