Nestor: Nested Sparsity for Scalable MoE Transformers

Nestor introduces a nested sparsity mechanism that hierarchically composes sparsity in both the attention and MoE operators. This method outperforms conventional MoE transformers at equivalent compute and sparsity levels in language modeling.