This meta-study organizes hallucination detection benchmarks along three axes: provenance, typology, and granularity. The analysis reveals that current benchmark comparisons are often invalid due to inconsistent definitions of what constitutes a hallucination.
HOW THIS AFFECTS YOU
●
researcherYou should choose your hallucination benchmarks carefully based on the specific error type you want to measure.
●
policyThis highlights the difficulty in creating standardized safety and accuracy benchmarks for LLMs.