Megadose The AI curve, compressed.

Lost in the Funnel: Selection-Induced Miscalibration in Ranking and Decision Systems

· ArXiv · AI/CL/LG ·
Recalibrating scores where decisions are made cut calibration error by 60–80% in a live production test.

The paper finds that filtering and reranking can make calibrated scores unreliable as probabilities downstream, even when calibration holds at the model boundary. Its diagnostic framework separates inherited model error from errors introduced by selection. The authors’ Funnel-Aware Recalibration adjusts scores for the selection policy at the decision boundary; a two-region A/B test also showed improved auction efficiency. The paper is under review, and the adapted public datasets and generation pipeline are planned for release.

ArXiv · AI/CL/LG's note

score 4

Categories: Research