Ablation Study of TALH Hybrid MLA-SSM Language Model Architecture

Ablation tests on the TALH hybrid model show that removing the SSM branch causes larger validation perplexity degradation than removing the MLA branch. In a 117--217M parameter setup, a dense-FFN hybrid achieved a lower PPL of 231 while using 3.87 GB less peak training memory than the top-2 ternary-MoE version.