Model Annealing Increases Accuracy via Self-Review and Consolidation
November 16, 2026
Model annealing improves top-1 accuracy for ConvNeXt, ViT, and MLP-Mixer by replacing late-stage cross-entropy training with a cycle of review and consolidation. The method uses a memory of past model states to average predictions, allowing models to learn from their own tempered knowledge without increasing inference costs.
HOW THIS AFFECTS YOU
●
builderYou can get better accuracy from your existing architectures without adding any extra latency during inference.
●
researcherYou can improve model performance on small datasets by treating remaining training budget as a consolidation phase.