Inverse Scaling in Vision-Language Models at Subtask Levels

This study identifies 375 instances of inverse scaling in VLMs, where larger models perform worse than smaller ones on specific subtasks. Across 558,334 evaluations, the findings show that model scale does not guarantee monotonic improvement across all multimodal question categories.