Same Output, Different Gold: Measuring How Reference Choice Moves a Multilingual Benchmark Score
Changing only the reference labels moved a fixed system’s benchmark score by up to 7.55 F1 points in one language.
The paper tests a six-language PII benchmark by keeping model outputs unchanged and swapping among retained annotation references. One output’s score shifted 4.95 F1 points; another shifted 2.00, with the system comparison also moving. The authors trace the effect to an undocumented aggregation default that often preserved one annotator’s labels when adjudication did not trigger, making the shipped gold partly copy an output being scored. They argue benchmarks should report a reference-sensitivity band alongside the score. HF Daily Papers' note
The paper tests a six-language PII benchmark by keeping model outputs unchanged and swapping among retained annotation references. One output’s score shifted 4.95 F1 points; another shifted 2.00, with the system comparison also moving. The authors trace the effect to an undocumented aggregation default that often preserved one annotator’s labels when adjudication did not trigger, making the shipped gold partly copy an output being scored. They argue benchmarks should report a reference-sensitivity band alongside the score. HF Daily Papers' note
score 4