LLM Performance Fluctuation in Data Quality Annotation Tasks
August 20, 2026
Evaluation of LLMs for e-commerce data quality shows zero-shot performance on entity matching (F1=0.948) nearly matches rule-based baselines (F1=0.950). Notably, few-shot prompting can actually degrade performance, highlighting the risks of using small validation samples for prompt tuning.
HOW THIS AFFECTS YOU
●
builderAvoid relying on small validation sets to optimize prompts for data labeling, as it may lead to performance regressions.
●
researcherThis underscores the unreliability of small-sample prompt evaluation in automated data quality pipelines.