Spurious Contexts Shift Model Ratings by up to 1.48 Points

Evaluation of seven frontier and open-weight models shows that irrelevant social contexts, such as teacher demographics or experience, can shift ratings by up to 1.48 points on a 7-point scale. The study highlights the inadequacy of current SFT and DPO methods in fully mitigating these biases in high-stakes decision-making scenarios.