Mixtures of SubExperts Mitigate Catastrophic Forgetting in Continual Learning
July 17, 2026
The MoSE framework addresses the stability-plasticity dilemma by augmenting transformer layers with lightweight, modular SubExperts and a learned routing function. This approach allows for localized knowledge storage and task-specific composition without the linear parameter growth of task-isolated expansions.
HOW THIS AFFECTS YOU
●
builderYou may be able to deploy models that learn continuously without overwriting previous capabilities.
●
researcherThis offers a new architectural method for structured parameter decomposition in LLMs.