ORCA-bench Evaluates LLM Agents on Production Oncall Tasks

ORCA-bench introduces a benchmark for assessing language model agents in production-fidelity oncall scenarios. It uses an OpenTelemetry-instrumented microservice system and 1,079 root cause analysis tasks involving metrics, logs, traces, and source code to measure reasoning over noisy telemetry.