SAUCE: Summary Analysis Using Conversation Entailment
SAUCE ties summary scores to verifiable facts so users can locate hallucinations and omissions.
The pipeline evaluates summaries of conversations across languages without a reference summary, measuring factual accuracy and coverage through entailment between the conversation and summary. The authors report that it detects information lost through transcription and translation errors and identifies broken summaries. Its fact-level explanations let developers investigate errors and users check accuracy in noisy conditions. Preliminary investigations show strong alignment with human judgment.
ArXiv · AI/CL/LG's note
The pipeline evaluates summaries of conversations across languages without a reference summary, measuring factual accuracy and coverage through entailment between the conversation and summary. The authors report that it detects information lost through transcription and translation errors and identifies broken summaries. Its fact-level explanations let developers investigate errors and users check accuracy in noisy conditions. Preliminary investigations show strong alignment with human judgment.
ArXiv · AI/CL/LG's note
score 4