[arXiv]score: 0.12
FoRA Cuts LoRA Parameter Budget by 75% With Minimal Accuracy Loss
May 29, 2026
Fisher scoring selects task-informative layers in a single pass (under 1% of training cost), then trains LoRA down-projections on the Stiefel manifold to preserve orthonormality. FoRA matches AdaLoRA accuracy within 0.7–0.8 points at one-quarter the parameter count, and outperforms LoRA and DoRA at half their budget across LLaMA, Qwen3, and Gemma families from 270M to 32B parameters.
cs.CL
HOW THIS AFFECTS YOU
●
builderYou can fine-tune LLaMA/Qwen3/Gemma models at significantly lower parameter counts without meaningful accuracy regression, reducing storage and serving costs for adapter-based deployments.
●
researcherThe Fisher-layer-selection plus Stiefel-manifold training combination shows super-additive gains, offering a new axis for PEFT research beyond rank reduction.