Do Halting Heads Really Save Compute in Looped Language Models?

Halting heads in looped language models fail to achieve significant compute savings when using standard stopping rules or batch processing. Training up to 1.3B parameters shows that unless a strong penalty favors early stops, models tend to run every iteration. Furthermore, batching sequences until the slowest token finishes negates the efficiency gains reported via average iteration counts.