Interpreting Language Model Hidden States at Scale

OmniLens scales trained lens methods to any model-width activation, including residual streams, attention, and MLP layers. Using low-rank translators, parameter growth becomes linear relative to model width, reducing trainable parameters by up to 98.4%. A Top-k Subset-KL training approach further decreases peak memory requirements by up to 70%.