Aggregate Accuracy Hides Diagnosis-Specific Failure in Medical Vision-Language Models
November 13, 2026
Auditing five open medical VLMs on ChestX-ray14 reveals that high aggregate accuracy masks zero-percent detection rates for specific diagnoses like pneumothorax. HuatuoGPT-Vision and Qwen2.5-VL-7B failed all 432 positive cases despite providing fluent, anatomically specific explanations that correctly identified lung collapse signs before denying their presence.