Verifiable Social Reasoning for LLM Assistants
The paper tests whether assistants can infer hidden social motives when the facts arrive through a user’s subjective account.
The authors introduce Fuse, a multi-agent simulation where a target agent has a hidden motive and a user-agent relays the situation to an LLM assistant. Because the motive is built into the simulation, the benchmark has verifiable ground truth. A human study with 24k annotations is used to validate the simulation’s faithfulness. Across 12 LLMs, the study finds that user mediation, biased framing, missing detail, and longer conversations can all complicate correct social reasoning.
ArXiv · AI/CL/LG's note
The authors introduce Fuse, a multi-agent simulation where a target agent has a hidden motive and a user-agent relays the situation to an LLM assistant. Because the motive is built into the simulation, the benchmark has verifiable ground truth. A human study with 24k annotations is used to validate the simulation’s faithfulness. Across 12 LLMs, the study finds that user mediation, biased framing, missing detail, and longer conversations can all complicate correct social reasoning.
ArXiv · AI/CL/LG's note
score 5