ReplayRCA Measures Agent Failure Diagnosis via Causal Repair
November 2, 2026
ReplayRCA introduces a benchmark that evaluates an agent's ability to diagnose and fix multi-step task failures using counterfactual replay. It scores performance based on a causal repair metric: the success rate of the agent continuing after a single corrective intervention in a deterministic environment.
HOW THIS AFFECTS YOU
●
builderYou can use this to move beyond simple agreement-based evaluation toward functional error correction testing.
●
researcherThis provides a more rigorous method for testing the diagnostic capabilities of autonomous agents.