Novel Claim or Déjà Vu? Rethinking "Contamination-Free'' Dynamic Evaluation for Multimodal Automated Fact-Checking
Post-cutoff claims still showed potential contamination in 17.09% to 29.30% of cases.
The paper argues that dynamic benchmarks reduce contamination risk but do not make multimodal fact-checking evaluations clean by default. The authors find that some newer claims can still be checked using public information available before a model’s cutoff, either directly or by combining older evidence. In their tests, contamination inflated Macro-F1 by up to 11.34 points and could change system rankings. The paper was accepted at ACM Multimedia 2026. HF Daily Papers' note
The paper argues that dynamic benchmarks reduce contamination risk but do not make multimodal fact-checking evaluations clean by default. The authors find that some newer claims can still be checked using public information available before a model’s cutoff, either directly or by combining older evidence. In their tests, contamination inflated Macro-F1 by up to 11.34 points and could change system rankings. The paper was accepted at ACM Multimedia 2026. HF Daily Papers' note
score 4