Linear Self-Attention Transformers Implement Closed Form Least Squares Regression

Transformers with linear self-attention can approximate the closed-form analytical solution to simple linear regression tasks. The mechanism utilizes layer normalization to achieve these estimates rather than the gradient descent algorithms typically observed in standard in-context learning research.