Causal Evidence Training Outperforms Attention Distillation in Sparse Routing

Research shows that training sparse attention selectors using causal evidence sets significantly outperforms distillation from dense teacher attention. In a two-step reference task, selectors trained on causal evidence achieved higher accuracy than those trained on attention patterns, which often capture irrelevant context.