Counsel Dataset Benchmarks LLM-as-Judge Reliability on Agentic Tasks

Counsel is the first public meta-evaluation dataset for agentic task assessment, containing process-level critiques from open-weight LLM judges on tau-bench and DA-Code benchmarks paired with human meta-evaluations of those critiques. It targets the reliability gap in LLM-as-judge pipelines used to scale agentic evaluation where human annotation costs hours per trajectory.