A Conditional No-Hair Theorem for Transformers: How Language Models Forget Their Prompt

Autoregressive transformer hidden states lose prompt-specific information following a power law decay $t^{-\alpha}$. Trained models demonstrate faster forgetting with $\alpha \approx 0.55$--$0.7$, while untrained models retain prompt influence longer at $\alpha \approx 0.2$--$0.5$. The mathematical bound for this decay scales as $O(P(\log t)^{L-1}/t)$ for finite-depth pre-LN softmax architectures.