Megadose The AI curve, compressed.

Nunchi: Ears Help Eyes in Multi-Party Conversations

· ArXiv · AI/CL/LG ·
Nunchi identifies whom someone is looking at with a reported 4.8 percentage-point gain over the strongest existing method.

The framework gives a vision-language model transcripts, gaze estimates and speaker cues, linking each cue to the same candidate people. The authors also introduce SORBET, a benchmark of 3,340 samples from 1,578 clips with synchronized speech, transcripts and labels identifying the person receiving attention. On SORBET, conversational cues improve results, while additional visual cues do not. The gains hold across three vision-language models, with automatic transcripts and on two existing datasets.

ArXiv · AI/CL/LG's note

score 4

Categories: Research