Mechanistic Interpretability of On-Policy Distillation via Swap Readout

Using sparse crosscoders and a new 'swap readout' method, this research analyzes how on-policy distillation (OPD) changes a student model's feature usage. The study tracks how internal representations evolve during the transfer of knowledge from a teacher.