Demographic Prompting Biases LLM Judges Toward Majority Groups
September 2, 2026
Testing 23 open-weight LLMs shows that sociodemographic prompting causes judges to align with specific groups rather than remaining neutral. Without demographic conditioning, models tend to reproduce the judgments of White, college-educated annotators.
HOW THIS AFFECTS YOU
●
researcherYou must account for inherent perspective biases when using LLMs for subjective evaluation.
●
policyThis finding demonstrates how LLM-based evaluation can unintentionally reinforce majority demographic norms.