Selection Bias in Chain-of-Thought Faithfulness Evaluation
October 28, 2026
Evaluating Chain-of-Thought (CoT) faithfulness only on correctly answered instances introduces a selection gap. Using a structural causal model, the authors identified a 7.4 percentage-point gap in GLM-5.3 performance, demonstrating that answer correctness acts as a confounder in faithfulness benchmarks.
HOW THIS AFFECTS YOU
●
researcherYou should account for correctness-induced selection bias when designing benchmarks to evaluate whether a model's reasoning actually matches its output.