MoME replaces single token embedding rows with a mixture of M slots, using a learned gate to select slots based on hidden state context. This prevents the collapse of different semantic meanings for the same token, such as 'python' the language versus the animal.
HOW THIS AFFECTS YOU
●
builderYou can improve model accuracy on polysemous tokens by using context-aware embedding retrieval.
●
researcherThis introduces a more efficient way to scale model capacity via conditional memory lookups.