Embedding-Based GUI Grounding Often Conflates Semantics with Label Recovery
August 25, 2026
Evaluation of GUI grounding shows that high embedding similarity frequently stems from simple visible-label recovery rather than semantic understanding. Lexical baselines remain competitive, and embedding-based performance is often predictable from label type and candidate pool size.
HOW THIS AFFECTS YOU
●
builderDo not rely on simple embedding similarity to validate GUI agentic capabilities; use lexical baselines.
●
researcherYou must stratify evaluations by label type to ensure you are measuring grounding, not just text matching.