Mutation Analysis Reveals Flaws in GPU-Kernel Benchmarks
September 1, 2026
Mutation analysis of 188 KernelBench problems shows that official checkers miss 16.9% of injected CUDA faults. The study provides a metric for assessing the adequacy of benchmark oracles used in LLM-generated GPU kernel evaluation.
HOW THIS AFFECTS YOU
●
builderYou should be cautious when using current kernel benchmarks for RL rewards, as they have high false-negative rates.
●
researcherThis provides a formal method for quantifying the reliability of code generation benchmarks.