LayerRoute: Adaptive Layer-Skipping via Joint LoRA Training
September 15, 2026
LayerRoute implements per-layer hard-gated routing and LoRA fine-tuning to enable adaptive transformer layer-skipping in Qwen2.5-0.5B-Instruct. The method achieves a 1.04x mean wallclock speedup while improving perplexity by approximately 1.1 through joint adaptation of the router and adapters.
HOW THIS AFFECTS YOU
●
builderYou can achieve inference speedups without sacrificing model quality by using joint router and LoRA training.
●
researcherYou can utilize straight-through estimators and gate-regularized objectives to stabilize skip-pattern convergence.