OmniSeek: Native Tool Integration for Multi-turn Audio-Visual Reasoning
OmniSeek turns an omni-model into an agent that chooses when to inspect audio or video instead of ingesting the whole sequence at once.
It retrieves short audio or visual windows during multi-turn reasoning, then feeds those segments back into context for the next step. The authors built OmniTraj-170K, a synthetic set of multi-hop trajectories with interleaved audio and visual evidence, to train that behavior. They then use reinforcement learning with verifiable rewards, including an objective meant to prevent the model from solving tasks through one modality alone. The paper reports consistent gains on audio-visual reasoning benchmarks. HF Daily Papers' note
It retrieves short audio or visual windows during multi-turn reasoning, then feeds those segments back into context for the next step. The authors built OmniTraj-170K, a synthetic set of multi-hop trajectories with interleaved audio and visual evidence, to train that behavior. They then use reinforcement learning with verifiable rewards, including an objective meant to prevent the model from solving tasks through one modality alone. The paper reports consistent gains on audio-visual reasoning benchmarks. HF Daily Papers' note
score 5