Gumbel Distillation Improves Parallel Text Generation Quality
July 24, 2026
Gumbel Distillation uses the Gumbel-Max trick to map latent noise to autoregressive teacher outputs, enabling parallel decoders to better model joint token distributions. On LM1B and OpenWebText, the method yielded a 30.0% MAUVE score improvement and 10.5% improvement in generative metrics.
HOW THIS AFFECTS YOU
●
researcherYou can use this model-agnostic technique to close the quality gap between non-autoregressive and autoregressive decoders.