Separating Representation from Reconstruction Enables Scalable Text Encoders

CrossBERT decouples representation learning from token reconstruction to address the misalignment found in traditional BERT architectures. By utilizing a complementary masking strategy with ratios exceeding 50%, the method increases throughput by 1.5x to 2x and doubles sample efficiency while improving MTEB and frozen GLUE benchmarks.