SMAT: Efficient Merge-Aware Training for Model Merging
September 26, 2026
SMAT optimizes experts by jointly minimizing task loss and expected loss at simulated merged parameters. It models common merging operations as scaling, masking, and perturbation to ensure better performance after weights are combined.
HOW THIS AFFECTS YOU
●
builderYou can merge multiple specialized models with higher confidence in the resulting performance.
●
researcherThis provides a formal framework for accounting for merging operations during the training phase.