Claimcheck Benchmark for Specification Faithfulness
October 14, 2026
Claimcheck is a benchmark designed to measure the gap between natural-language requirements and formal artifacts in four languages: Dafny, Lean 4, Isabelle, and Rocq. It contains 3,234 rows to evaluate if LLM-generated specifications actually capture intended meaning.
HOW THIS AFFECTS YOU
●
researcherYou can use this to evaluate the reliability of LLMs in formal verification tasks.
●
policyThis highlights the safety risks of using LLMs to generate specifications that may not match human intent.