CAST Framework for Cross-Tokenizer On-Policy Distillation
October 12, 2026
CAST solves the credit alignment problem in cross-tokenizer distillation by redistributing chunk-level advantage signals across student tokens. It uses predictive uncertainty and teacher-signal smoothness to weight tokens, effectively solving a KL-regularized credit-allocation problem for unaligned sequence segments.
HOW THIS AFFECTS YOU
●
researcherYou can use this method to improve distillation efficiency when teacher and student models use different tokenizers.