Omega-S Penalty for Reducing Catastrophic Forgetting in LLM Fine-Tuning

Omega-S is a low-overhead fine-tuning regularizer that uses only the weight matrix to prevent knowledge degradation. It requires no previous-task data or Fisher matrices, adding under 4% to step cost and significantly improving HumanEval retention on Llama-3-8B.