PinyinBench Evaluates Phonological Knowledge in Multimodal Language Models

PinyinBench evaluates how LMs represent Chinese phonology, revealing that while DS-v4-flash scores 90%+ on homophone judgment, accuracy falls to 72% when characters differ only by lexical tone. Probing shows tone is decodable early in embeddings, whereas segmental information requires greater model depth to become decodable.