This method reorganizes trainable PEFT parameters into square matrices for recursive low-rank decomposition. When applied to LoRA, DoRA, MoSA, and RaSA on LLaMA-3.1-8B and Qwen2.5-7B, it drastically reduces the number of trainable parameters while maintaining performance on commonsense and math reasoning benchmarks.
HOW THIS AFFECTS YOU
●
builderYou can reduce memory and compute requirements for fine-tuning large models on constrained hardware.
●
researcherYou can achieve higher parameter efficiency by applying recursive low-rank decomposition to existing PEFT architectures.