Gibbs-Gap Policy Optimization for High-Variance Off-Policy Reinforcement Learning

GGPO minimizes a log-mean-exp contrast of KL-adjusted rewards to stabilize reinforcement learning when training on responses from stale behavior policies. The method eliminates the need for learner-to-behavior importance weights, reducing variance in long-sequence response training.