MetaKernelBench: Measuring GPU Kernel Knowledge Transfer Beyond Code
MetaKernelBench tests whether a GPU-kernel agent’s distilled “skill” actually helps when the same problem is attempted in a different DSL.
The benchmark uses 74 fused-subgraph problems, each paired between CuTe DSL and TIRx with only the DSL changed. Agents first try a variant alone, distill a natural-language skill, then use only that skill as extra input for the paired attempt in the other DSL. Across six models, cross-DSL lift ranges from -19% to +29%, with regressions on 16% to 45% of problems in every tested model and direction. The authors find transfer depends more on whether the source attempt outperformed the target’s solo attempt than on whether the source passed verification. HF Daily Papers' note
The benchmark uses 74 fused-subgraph problems, each paired between CuTe DSL and TIRx with only the DSL changed. Agents first try a variant alone, distill a natural-language skill, then use only that skill as extra input for the paired attempt in the other DSL. Across six models, cross-DSL lift ranges from -19% to +29%, with regressions on 16% to 45% of problems in every tested model and direction. The authors find transfer depends more on whether the source attempt outperformed the target’s solo attempt than on whether the source passed verification. HF Daily Papers' note
score 5