Marginal Attention Space Reveals Conserved Text-Intrinsic Signals Across LLMs

By marginalizing over query positions in post-softmax attention weights across 60+ LLMs, researchers identified a text-intrinsic signal that is robustly conserved across different models. This marginal attention space is theoretically linked to the input-output Jacobian of the network.