Best Practice Critic Optimization (BPCO) improves the stability of critic-based reinforcement learning by using DPPO, bounded value predictions, and Monte Carlo targets. It allows the critic to be conditioned on hidden reference information, such as grading rubrics, during training.
HOW THIS AFFECTS YOU
●
researcherYou can more reliably use critics to estimate token-level advantages in group-based RL methods like GRPO.