The Attention Triangle in Audio-Video Models
The paper says audio-video models can reroute meaning across modalities, causing “canonical” but wrong generations.
The authors focus on the cross-attention links between text, audio, and video streams. They find the audio-video link works both ways: sound can shape visuals, and visuals can shape sound. When a prompt conflicts with the model’s learned priors, that link can override the intended conditioning and leak semantics into the wrong place. They use attention-derived signals to diagnose the routing and guide inference-time fixes for better cross-modal alignment. HF Daily Papers' note
The authors focus on the cross-attention links between text, audio, and video streams. They find the audio-video link works both ways: sound can shape visuals, and visuals can shape sound. When a prompt conflicts with the model’s learned priors, that link can override the intended conditioning and leak semantics into the wrong place. They use attention-derived signals to diagnose the routing and guide inference-time fixes for better cross-modal alignment. HF Daily Papers' note
score 5