LLM Annotation Often Follows Label Names Instead of Provided Definitions
November 9, 2026
A counterfactual audit across three political datasets shows models rarely adhere to researcher-supplied category definitions, instead relying on prior associations with label names.
HOW THIS AFFECTS YOU
●
builderWhen using LLMs for data labeling, use k-nearest-neighbor retrieval to improve performance.
●
researcherYou cannot trust high accuracy scores as proof that a model understands your specific taxonomy.