On-Policy Mid-Training Distillation for specialized LLM adaptation
November 11, 2026
On-Policy Mid-Training Distillation (OPMD) replaces standard next-token prediction with query-driven knowledge acquisition during continual pre-training. This prevents the performance degradation seen in models like Qwen3-1.7B-Base when using standard imitation on specialized corpora.
HOW THIS AFFECTS YOU
●
builderYou can use this method to adapt smaller models to new domains without breaking their existing knowledge.
●
researcherThis replaces offline text imitation with active trajectories and dense token-level rewards from a frozen teacher.