Learnable Subspace Projections (LSP) replace local, error-based compression criteria with an end-to-end learned approach using orthogonal projectors. This method mitigates error compounding across deep layers, preventing performance collapse at high compression ratios.
HOW THIS AFFECTS YOU
●
builderYou can achieve higher compression ratios for transformers without the typical accuracy drop-off.
●
researcherThis offers a more robust way to optimize subspaces for weight factorization.