An empirical study using the Embodied Agent Arena finds that while closed-weight VLMs lead in geometry, spatial reasoning, and manipulation, open-weight models struggle with task planning. The study utilizes GeoProbe to evaluate camera, motion, and depth estimation across 1,000 cases.
HOW THIS AFFECTS YOU
●
builderBe cautious when using open-weight VLMs for high-level robot task planning.
●
researcherNote the significant performance gap between closed and open-weight VLMs in robotics tasks.