Words to World: Benchmarking LLMs in Interactive Physical Environments

Words to World introduces a closed-loop benchmark for evaluating LLMs through direct control of a hardware harness involving robotic arms, cameras, and sensors. Unlike static text evaluations, this framework measures a model's ability to interpret real-time physical feedback and adapt instruction sequences based on sensory state changes.