Deconstructing Off-Policy Ratios: Entropy-Scaled Trust Regions for Asynchronous Reinforcement Learning

Title: Deconstructing Off-Policy Ratios: Entropy-Scaled Trust Regions for Asynchronous Reinforcement Learning Source: arxiv Asynchronous reinforcement learning stability suffers because importance ratio scales vary systematically with token entropy. Low-entropy tokens amplify sampling noise during stale data training, while high-entropy tokens exhibit legitimate exploratory deviations. Applying entropy-based scaling to importance ratios prevents policy collapse by adjusting correction thresholds according to local token uncertainty.