Enabling Token-wise Importance Sampling in Off-Policy RLVR with Proximal KL Regularization

Token-wise importance sampling in RLVR avoids multiplying likelihood ratios across full responses by using proximal KL regularization to maintain unbiased gradients. This method ensures the exact proximal optimizer remains the unique stationary policy for mixtures of older policies without requiring a fitted critic or full-response weights.