Value Flattening Failure Mode Identified in PPO Critics

Research identifies Value Flattening in PPO, where critic predictions remain flat despite sharp changes in true state values across intermediate states. This phenomenon, linked to implicit variance penalties and redundant updates, worsens as state space scale increases.