SCPO Reward Model Training Boosts Minority Culture Preferences by 7 Points

Steerable Cultural Preference Optimization (SCPO) trains reward models to balance diverse cultural preferences without over-indexing on majority annotator groups. Evaluated on PRISM and GlobalOpinionQA across 7 countries, it improves minority reward model performance by up to 7 points and is up to 280% more training-data-efficient than the baseline.