Transformers Develop Internal State Representations for Tower of Hanoi

Mechanistic interpretability shows small Transformers trained on solution traces develop a geometrically faithful Sierpinski triangle representation of the Tower of Hanoi state space. Testing on frontier models like Qwen3.6-27B reveals that while large reasoning models solve puzzles, they struggle with complex state transitions.