CoT-Prefix Scoring Causes Massive Accuracy Drops in Vision-Language Models

Reasoning instructions can distort VLM evaluation by causing probability mass to move toward continuation tokens rather than answer labels. On ScienceQA, Qwen2.5-VL-7B accuracy dropped from 80.76% to 45.48% due to this scoring mismatch, despite answer information remaining in late layers.