Confidence Comes from Experience: Experiential Confidence Estimation from Reasoning to Agents
XConf estimates model confidence by checking how similar past attempts actually turned out.
The paper proposes storing graded prior episodes with the task, reflection, stated confidence, outcome, and a later lesson. For a new task, XConf retrieves comparable past cases, uses their success rate, then asks the model to revise its confidence in light of recurring failure modes. Across nine benchmarks and four models, it matched or beat ten-sample self-consistency on AUROC in 23 of 24 comparisons, with lower calibration error and far less generation cost. In selective prediction, dropping the 10% least-confident agent episodes raised delivered success by up to 8.7 points. HF Daily Papers' note
The paper proposes storing graded prior episodes with the task, reflection, stated confidence, outcome, and a later lesson. For a new task, XConf retrieves comparable past cases, uses their success rate, then asks the model to revise its confidence in light of recurring failure modes. Across nine benchmarks and four models, it matched or beat ten-sample self-consistency on AUROC in 23 of 24 comparisons, with lower calibration error and far less generation cost. In selective prediction, dropping the 10% least-confident agent episodes raised delivered success by up to 8.7 points. HF Daily Papers' note
score 5