[arXiv]score: 0.14

SAE Feature Dictionaries Outperform EWC for LLM Continual Learning

June 26, 2026

Regularizing in activation space using pretrained Sparse Autoencoders as a monosemantic feature dictionary addresses the polysemanticity problem that makes weight-space methods like EWC underperform on LLMs. The derived loss function subsumes EWC as a special case and avoids storing replay data.

HOW THIS AFFECTS YOU

●

builderIf you are fine-tuning LLMs sequentially on new tasks, SAE-guided activation regularization is a drop-in alternative to EWC that avoids replay buffers and better preserves pretrained knowledge.

●

researcherThe theoretical derivation showing EWC as a special case of the SAE-regularized objective gives a principled basis for comparing continual learning methods on LLMs.

read original ↗arxiv.org

← back to feed