Random Metadata Cues Impact LLM Social Inference Accuracy
August 7, 2026
A randomized audit reveals that while verified survey-country metadata reduces Brier loss by 0.040, assigning random country labels causes LLMs to shift forecasts by 0.214. The study demonstrates that LLMs are highly susceptible to spurious country-directed cues even when the labels are disclosed as random.
HOW THIS AFFECTS YOU
●
researcherYou should account for spurious metadata biases when using LLMs for social science forecasting.
●
policyThis highlights vulnerabilities in how LLMs process demographic and geographic information during inference.