Evaluation Revisited: A Taxonomy of Evaluation Concerns in Natural Language Processing

Researchers published a scoping review on arXiv introducing a structured taxonomy of evaluation concerns in NLP, consolidating decades of methodological critique predating the LLM era. The work synthesizes recurring failure modes across evaluation design and interpretation, delivering a practitioner-facing checklist for more rigorous benchmark construction. ML engineers building or auditing LLM evaluation pipelines should treat this as a foundational reference, particularly given widespread issues like benchmark contamination, metric misalignment, and construct validity gaps.