SFT-as-Context Reduces Forgetting in Specialized LLM Fine-Tuning

SFT-as-context is a training-free method that prevents catastrophic forgetting by using a fine-tuned model's response as context for the original parent model. In testing, it maintained general capabilities within 2.2 percentage points of the parent model while remaining competitive on specialized benchmarks like AIME 2024.