LRCC Improves Llama-2-7B Accuracy via Conditional Low-Rank Compression

Low-Rank Conditional Computation (LRCC) uses lightweight routers to select between nested low-rank paths based on input tokens. On Llama-2-7B, this method achieves a 7.6 percentage-point increase in downstream accuracy compared to static low-rank methods while maintaining the same average active-parameter budget.