Can LLM Annotators Match Human Consistency? Benchmarking Human Agreement and Optimizing Multi-Agent with Contrastive Knowledge

The ANT-A benchmark introduces 7 datasets across multiple modalities using 5 independent human labels per instance to evaluate distribution-aware agreement via Fleiss kappa. The accompanying ANT-OR framework utilizes multi-agent distillation to align model annotations with these human labeling distributions, addressing inconsistencies found in single-label datasets.