Asymmetric Modality Biases in Vision-Language Model Memory Conflicts

Vision-language models exhibit asymmetric biases when in-context information conflicts with parametric memory, favoring text for textual entities and parametric training data for visual entities. This stems from late representational alignment and longer processing times for visual entities, which prevents the suppression of factual recall.