What Does an Evaluation License? A Commit-Bound Census of Claim-Relative Inference in Inspect Evals
The paper says most Inspect Evals units cannot support deterministic claim replay from the available historical and semantic evidence.
Xi Qin formalizes a “claim-replay” layer for evaluation artifacts, separating a reported metric from the claim attached to it. The audit covers 124 mechanically eligible Inspect Evals units at a pinned commit. It gives every unit a terminal disposition, with 110 stopping before deterministic inference because required evidence or grounding is missing. Where execution does close, the paper reports exact values, winners, rankings, and pairwise relations by claim resolution and family. HF Daily Papers' note
Xi Qin formalizes a “claim-replay” layer for evaluation artifacts, separating a reported metric from the claim attached to it. The audit covers 124 mechanically eligible Inspect Evals units at a pinned commit. It gives every unit a terminal disposition, with 110 stopping before deterministic inference because required evidence or grounding is missing. Where execution does close, the paper reports exact values, winners, rankings, and pairwise relations by claim resolution and family. HF Daily Papers' note
score 4