SG-Blend learns an interpolation between a parametric Swish variant (SSwish) and GELU using a per-layer blend coefficient and two additional scalars per FFN block. This allows each transformer layer to adapt its activation shape to resolve gradient pathologies caused by LayerNorm.
HOW THIS AFFECTS YOU
●
builderIntegrating this could improve transformer training stability by mitigating gradient issues in deep FFN blocks.
●
researcherThis method provides a low-overhead way to optimize activation functions for specific architecture depths.