[arXiv]score: 0.15

DiagFlowBench: 1,676 Conversations Test LLM Abstention in Industrial Diagnostics

June 17, 2026

Ten commercial and open-weight LLMs evaluated on 50 industrial diagnostic flowcharts show high variability in abstention rates when queries go off-procedure, with models frequently selecting a plausible but wrong step rather than refusing to answer. The benchmark highlights that grounding to documentation does not reliably prevent contextually mismatched but authoritative-sounding responses.

HOW THIS AFFECTS YOU

●

builderIf you are building RAG or grounded advisory systems for industrial or maintenance use cases, this benchmark exposes a specific failure mode — mapped-but-wrong responses — that abstention metrics alone won't catch.

●

researcherThe dataset formalizes out-of-scope handling as a distinct evaluation axis separate from hallucination, useful for benchmarking grounding strategies.

read original ↗arxiv.org

← back to feed