Training 12M Parameter Portuguese Language Models on Single GPUs
October 30, 2026
A new tutorial and Python package demonstrate training autoregressive and masked diffusion language models from scratch on a single consumer GPU. For 11M to 12M parameter models in Brazilian Portuguese, masked diffusion improved up to 256 epochs and exceeded autoregressive likelihood by 0.06 bits per byte.
HOW THIS AFFECTS YOU
●
builderYou can use these tools to train domain-specific small language models on minimal hardware.
●
researcherThe comparison protocol provides a baseline for evaluating diffusion versus autoregressive generation in small-scale models.