Measuring Non-Additive Behavior in Sparse Autoencoder Features

Sparse autoencoders (SAEs) often fail to exhibit purely additive behavior in downstream tasks. The researchers introduce SAE Additivity Residuals (SAR) to quantify unexplained interaction effects, demonstrating that learned SAE coordinates provide better compositionality than random subspace rotations.