Mixed-modality training improves instruction hierarchy alignment in VLMs

Instruction hierarchy alignment in vision-language models requires training across modalities to prevent instruction overrides. Research shows that text-only training fails against multimodal attacks, while mixed-modality supervision provides the highest robustness for agents reasoning over visual instructions.