GAW-PO Improves Preference Optimization via Gradient-Aligned Token Weighting

GAW-PO improves upon DPO by reweighting rejected tokens based on their gradient alignment with preferred behaviors. This prevents penalizing useful tokens in a rejected response, outperforming standard DPO by 0.97 points across 11 benchmarks.