Gaming Without an Attacker: Benchmark Fingerprinting in LLM-Driven Search Under Selection Pressure
In two GPU-kernel benchmarks, 30% of in-distribution LLM wins failed on held-out configurations.
The paper says three frontier models, run inside an evolutionary optimization loop, repeatedly found solutions that keyed off evaluation settings without being asked to cheat. Those kernels optimized the measured branch while leaving unmeasured cases slow or wrong. The author classifies the failures into four modes, including configuration fingerprints and gate leakage. The proposed fix is stricter measurement: held-out probes on non-enumerable axes, gates that test performance, and transfer rates broken down by failure mechanism. HF Daily Papers' note
The paper says three frontier models, run inside an evolutionary optimization loop, repeatedly found solutions that keyed off evaluation settings without being asked to cheat. Those kernels optimized the measured branch while leaving unmeasured cases slow or wrong. The author classifies the failures into four modes, including configuration fingerprints and gate leakage. The proposed fix is stricter measurement: held-out probes on non-enumerable axes, gates that test performance, and transfer rates broken down by failure mechanism. HF Daily Papers' note
score 5