DeltaMomentum uses delta rule for anisotropic momentum updates
August 21, 2026
DeltaMomentum replaces standard exponential moving average momentum with a key-value based update rule. By utilizing the delta rule, the optimizer allows each gradient direction to be forgotten at a rate proportional to its frequency in the input data.
HOW THIS AFFECTS YOU
●
researcherYou can use this method to improve training efficiency in networks with highly anisotropic input distributions.