Megadose The AI curve, compressed.

Black-Box Uncertainty Quantification for Large Language Models via Ensemble-of-Ensembles

· ArXiv · AI/CL/LG ·
Repeated answers and reworded prompts let researchers estimate prompt ambiguity separately from model knowledge gaps, using only black-box access.

The method samples multiple answers to each prompt, then repeats the process with meaning-preserving rewordings and measures variation in an embedding space. Across five short-answer benchmarks and four instruction-tuned models, the authors report error-ranking performance matching leading black-box methods at the same generation budget. Controlled tests found the two uncertainty estimates responded selectively to prompt ambiguity and knowledge gaps. A preliminary long-answer test extended the approach to claim-level factuality without retraining.

ArXiv · AI/CL/LG's note

score 5

Categories: Research