Contamination Filtering Crucial for Hindsight-Guided Distillation in Rare Disease Diagnosis
October 5, 2026
A 1.5B student model fine-tuned on 8B teacher reasoning traces shows that performance gains in rare disease diagnosis stem from contamination filtering rather than hindsight distillation alone. Unfiltered students suffer from GT hallucination, incorrectly embedding ground-truth phrases into reasoning chains 33.9% of the time.
HOW THIS AFFECTS YOU
●
researcherThis highlights a critical failure mode where SFT copies label-visible artifacts during the distillation process.
●
healthYou must be cautious of reasoning traces that appear correct but are actually artifacts of the training data.