TranSGrid Testbed Reveals Systematic Generalization Gaps in Transformers

The TranSGrid testbed combines deductive, inductive, and abductive reasoning to evaluate systematic generalization. Experiments show a significant performance drop in seven Transformers, where the largest model fell from 79.6% accuracy on held-out test sets to 55.3% on the reasoning-centered TranSGrid task.