SyntaxBench: Statistical Framework for Character-Level LLM Reasoning

SyntaxBench introduces a diagnostic framework to evaluate character-level reasoning through tasks like edit distance, palindrome detection, and substring extraction. The benchmark tests models from 2B to 32B parameters across various zero- to four-shot prompting configurations.