Auditability and stability analysis of Vals Finance Agent Benchmark v2
November 16, 2026
An analysis of the Vals Finance Agent Benchmark v2 shows that identical runs vary by up to 17% in rubric scoring. The study classifies failures across instructions, tools, and models, noting that an LLM analyzer agreed with human readers on fault diagnosis in 94% of cases.
HOW THIS AFFECTS YOU
●
builderYou should expect up to 17% variance in agent performance scores even with identical runs.
●
researcherThe study provides a taxonomy for diagnosing why financial agents fail.