Mid-Block Efficient Tuning Outperforms LoRA in Supervised Fine-Tuning
August 27, 2026
Analysis of SFT across 1B-32B parameter models shows instruction-following capabilities emerge primarily in middle layers. Mid-Block Efficient Tuning targets these intermediate layers, outperforming standard LoRA by up to 10.2% on GSM8K for OLMo2-7B.
HOW THIS AFFECTS YOU
●
builderYou can achieve better fine-tuning performance with lower parameter overhead by targeting specific model blocks.
●
researcherThis reveals that effective alignment is architecturally localized in middle layers rather than distributed.