Syntax Heads in Transformers Show Limited Causal Role in Predictions

Research indicates that while Transformer attention heads specialize in syntactic relations, they may not be causally responsible for syntactic predictions. Interventions on these heads had limited impact on model behavior, suggesting syntactic processing is more distributed.