In a study of cyberbullying severity, researchers found that only 4.1% of comments had unanimous agreement among five annotators. They demonstrate that predicting the full distribution of annotator judgments, rather than a single majority vote, preserves critical information about subjective disagreement.
HOW THIS AFFECTS YOU
●
researcherYou can capture nuance in subjective datasets by training models to predict label distributions instead of majority votes.
●
policyThis matters when building safety tools for subjective topics like harassment, where consensus is rare.