Sample-Level Auditing Reveals Heterogeneity in LLM Benchmarks

A new meta-evaluation framework audits benchmarks like MMLU and TruthfulQA at the individual sample level across five dimensions. This reveals that aggregate scores obscure significant variation in reasoning depth, ethics, and task properties.