Dispersion Loss Mitigates Embedding Condensation in Small Language Models

Dispersion loss is introduced to counteract embedding condensation, a phenomenon observed more severely in smaller language models than in larger ones. This technique aims to improve representational density and model scaling efficiency.