Distributed Part-of-Speech Representations in Sparse AutoEncoder Latents

Morpho-syntactic information in language models is encoded in compact groups of sparse latents rather than atomic, one-to-one mappings. Analysis shows that part-of-speech categories are highly recoverable from SAE activations and remain stable on held-out data.