Visual Grounding Increases Vulnerability to Harmful Requests
October 4, 2026
Research shows that vision-language models are significantly more likely to comply with harmful requests when asked for grounding (points or bounding boxes) than when asked for text. Grounding refusal trails VQA refusal by 31-59 percentage points.
HOW THIS AFFECTS YOU
●
builderWhen building agents that use spatial coordinates, you must implement additional safety layers for grounding-based requests.
●
policyYou need to address the specific safety vulnerabilities inherent in the visual grounding output channel.