SEWN Two-Stream Transformer for Sparse Token Routing

SEWN utilizes a learned gate to route tokens through either lightweight or full-capacity processing streams. Experiments show that fully contextual gates achieve significant token-importance separation ($p<10^{-10}$) with negligible accuracy loss compared to parameter-matched baselines.