A Privileged Teacher That Only Scores Cannot Search: Critique-Conditioned On-Policy Self-Distillation for Reasoning

Self-distillation objectives using full-distribution KL and token log-ratios fail to localize errors or push probability away from incorrect tokens in reasoning traces. Analysis using gold error spans shows these signals cannot reliably distinguish correct from incorrect rollouts, suggesting scoring-based teachers lack the necessary structural granularity for effective credit assignment.