Spurious Correlation Learning in Preference Optimization: Mechanisms, Consequences, and Mitigation via Tie Training

Analysis of spurious correlation learning in preference optimization methods like DPO, characterizing mechanisms of sycophancy and length bias with a provable mitigation strategy via tie training.