LoRA and Weight Decay (2023)

LoRA fine-tuning implicitly regularizes adapted weights toward the frozen base model due to weight decay interaction, meaning it solves a different optimization problem than full fine-tuning. Practitioners using LoRA as a drop-in replacement should account for this bias, particularly when the target task distribution diverges significantly from the pretrained model.