Unsupervised On-Policy Self-Distillation via Internal Consistency
August 8, 2026
U-OPSD enables LLM post-training using only a model's own generations. The method constructs pseudo-solutions through majority voting under a self-consistency threshold, conditioning the distribution on these solutions to distill knowledge without external ground-truth or teacher models.
HOW THIS AFFECTS YOU
●
builderThis offers a way to improve model performance in-house using only your existing model's outputs.
●
researcherYou can explore a new path for distillation that removes the dependency on massive teacher models.