Measuring the Checker: Mutation Analysis for GPU-Kernel Benchmark Oracles
The paper says KernelBench’s official checker misses 16.9% of witnessed injected faults in verified CUDA kernels.
The authors use mutation analysis to score GPU-kernel benchmark oracles, injecting 10,303 compilable faults across 188 problems. Precision faults are the largest blind spot: 78.6% escape the official check, versus 8.7% of arithmetic faults. They also find a published fuzzing recipe rejected correct kernels 107 times. Their optimized suites reach 98.0% detection with two inputs per problem, and they release the dataset as KernelBench-M. HF Daily Papers' note
The authors use mutation analysis to score GPU-kernel benchmark oracles, injecting 10,303 compilable faults across 188 problems. Precision faults are the largest blind spot: 78.6% escape the official check, versus 8.7% of arithmetic faults. They also find a published fuzzing recipe rejected correct kernels 107 times. Their optimized suites reach 98.0% detection with two inputs per problem, and they release the dataset as KernelBench-M. HF Daily Papers' note
score 6