swyx argues current Transformer and gradient descent stack is inefficient
November 21, 2026
swyx claims the combination of Transformers, gradient descent, and token-space reasoning is not among the 50% most efficient learning methods. He suggests future architectures will diverge from current pretraining and on-the-fly adaptation approaches.
HOW THIS AFFECTS YOU
●
researcherKeep an eye out for non-Transformer architectures that move beyond standard gradient descent.
●
founderCurrent scaling laws might not be the only path to efficient intelligence.