Knowledge Distillation Asymmetrically Impacts Model Bias

Knowledge distillation in small models shows asymmetric effects: it improves context-following in unambiguous tasks but destroys refusal calibration in ambiguous ones. In tests with SmolLM2 and OLMo-2, distillation caused models to provide stereotype answers instead of correctly abstaining.