ModernBERT Encoders for Icelandic scaling from 12M to 1B parameters
October 28, 2026
ModernBERT-based Icelandic encoders trained on 9B tokens outperform IceBERT across eleven tasks. A 1B parameter model warm-started from the English Ettin encoder achieves the highest scores across all benchmarks, demonstrating the efficacy of cross-lingual initialization for low-resource languages.
HOW THIS AFFECTS YOU
●
builderYou can integrate high-performing monolingual Icelandic encoders into your NLP pipelines.
●
researcherYou can leverage ModernBERT architectures and cross-lingual warm-starting to improve low-resource language performance.