Mechanistic Analysis Shows Circuit Collapse in Distilled Language Models

Mechanistic studies of DistilGPT-2 reveal a compression-collapse dichotomy where the Indirect Object Identification circuit suffers total collapse while the Greater-Than circuit survives via head remapping. This demonstrates that circuit compressibility dictates what knowledge persists after depth-reduction distillation.