Attention-Aware Routing (AAR) improves MoE performance by augmenting the router with temporal and spectral features from attention weights. In testing on OLMoE, it improved GSM8K scores by 3.37 percentage points without updating the base transformer.
HOW THIS AFFECTS YOU
●
builderThis technique allows for performance gains in MoE models through parameter-efficient routing updates.
●
researcherYou can improve MoE routing efficiency by coupling it with the model's existing attention circuits.