AI Consilium: a Benchmark of LLM Clinical Multidisciplinary Second Opinions
Model choice mattered more than report format in tests of clinical second opinions.
The benchmark used 300 resolved NEJM–MedQA cases, reserving 190 for held-out evaluation, with a physician-calibrated LLM judge scoring 3,040 final reports. Structured reports scored higher overall in seven of eight model pipelines, with the clearest advantage in management. Multidisciplinary synthesis improved all five shared criteria over the initial structured opinion, especially problem prioritization, factual grounding and reasoning coherence. GPT-5.6-Sol led every evaluated component and reached consensus in the fewest review rounds.
ArXiv · AI/CL/LG's note
The benchmark used 300 resolved NEJM–MedQA cases, reserving 190 for held-out evaluation, with a physician-calibrated LLM judge scoring 3,040 final reports. Structured reports scored higher overall in seven of eight model pipelines, with the clearest advantage in management. Multidisciplinary synthesis improved all five shared criteria over the initial structured opinion, especially problem prioritization, factual grounding and reasoning coherence. GPT-5.6-Sol led every evaluated component and reached consensus in the fewest review rounds.
ArXiv · AI/CL/LG's note
score 4