Impact of Privileged Information on On-Policy Self-Distillation

Using the AMPLE-Math suite, this study finds that reference-free distillation contributes most to performance gains in reasoning models like Qwen3-1.7B. The benefit of providing a teacher model with extra 'privileged' information or worked solutions is found to be relatively modest.