Contrastive Policy Optimization for Correctness-Aware RLVR
July 17, 2026
Contrastive Policy Optimization (CPO) replaces entropy-based advantage shaping with token-level contrastive disagreement between reference-guided and vanilla distributions. This method distinguishes useful uncertainty from confusion, outperforming standard RLVR methods in correctness signaling and generalization.
HOW THIS AFFECTS YOU
●
builderThis improves the efficiency of training models with verifiable reward signals.
●
researcherYou can use token-level disagreement as a more reliable signal for reinforcement learning from verifiable rewards.