Pinocchio: Fast Uncertainty Estimates for Black-Box Language Models
Pinocchio estimates whether a black-box LLM answer is likely correct without needing logits, weights, or fine-tuning access.
The paper presents an external calibrator for API-only models that may not expose internals. Trained on responses from seven LLMs, it reports 0.862 AUROC on held-out responses from those models. The authors say it transfers zero-shot to thirteen unseen models across eight organizations. A lightweight text-only 0.8B checkpoint matches their largest model’s AUROC, and they say the integration takes two lines of code. ArXiv · AI/CL/LG's note
The paper presents an external calibrator for API-only models that may not expose internals. Trained on responses from seven LLMs, it reports 0.862 AUROC on held-out responses from those models. The authors say it transfers zero-shot to thirteen unseen models across eight organizations. A lightweight text-only 0.8B checkpoint matches their largest model’s AUROC, and they say the integration takes two lines of code. ArXiv · AI/CL/LG's note
score 5