Measuring how LLM annotation changes with different definitions
November 11, 2026
This study introduces definition sensitivity to measure if LLMs follow provided rules or rely on pre-trained biases during social science annotation. By testing models with varying definitions of concepts like hate speech, the research quantifies disagreement between annotation sets.
HOW THIS AFFECTS YOU
●
builderYou should test your annotation prompts with multiple definitions to ensure your labels are consistent and not sensitive to wording.
●
researcherYou can use this method to check if your LLM-based annotator is actually following your rules or just using its own biases.