GPT-2 Reconstruction Reveals Architectural Bias Toward Information Locality

Fine-tuning GPT-2 models on impossible languages demonstrates that recovered natural language exhibits shorter dependency lengths than the original source. This reconstruction framework identifies a quantitative architectural bias toward information locality that is not captured by standard learnability experiments.