Cracks in the Foundation: Seemingly Minor Architectural Choices Impact Long Context Extension

Four specific architectural variations in dense transformers, used across Olmo, Llama, and Qwen families, compound to reduce long-context performance by up to 47%. These design choices are undetectable via short-context loss but become visible when applying context extension early in pretraining.