Multi-Task On-Policy Distillation Using Soft-Prompt Privileged Context
July 22, 2026
This method enables a single student model to learn from multiple task-specific teachers via learnable soft prompts while keeping the backbone frozen. Testing on Qwen3-1.7B and Phi-4-mini shows the student absorbs knowledge from K teachers in parallel without the representational drift typical of weight fine-tuning.
HOW THIS AFFECTS YOU
●
builderYou can use this to expand a small model's capabilities across multiple tasks using a single backbone.
●
researcherYou can use soft prompts to provide privileged context during distillation without inducing post-hoc rationalization.