Tampered ML Benchmark Reveals LLM Unreliability in Evaluating ML Research

Tampered ML tests an LLM's ability to detect methodological flaws in machine learning research proposals through 1,858 counterfactual pairs. Frontier models struggle with reliability; Opus 5 achieved only 77.9% accuracy and showed high sensitivity to the order of presentation.