Audit Reveals Failure Modes in AppWorld and WorkArena Task Verifiers
October 8, 2026
Mutation testing shows that execution-based verifiers in AppWorld and WorkArena frequently fail to catch incorrect agent actions. In AppWorld, verifiers accepted incorrect task variants in 6/15 cases, while WorkArena verifiers incorrectly passed 23 instances of non-default persisted values.
HOW THIS AFFECTS YOU
●
researcherYou should account for verifier blind spots when evaluating agentic performance.
●
policyThis highlights reliability risks in automated agent evaluation frameworks.