Contrastive Policy Optimization (CPO) uses token-level disagreement between reference and vanilla distributions to improve advantage shaping in RL. This method distinguishes useful uncertainty from detrimental confusion, addressing the zero-advantage problem in RLVR.
HOW THIS AFFECTS YOU
●
researcherYou can implement CPO to improve the reliability of verifiable reward training in RL models.