Policy Gradient for Counterfactual Invariance in LLMs

Standard alignment techniques like RL can inadvertently amplify counterfactual bias by increasing response quality differences across demographics. The authors propose a policy gradient method using group-level reward functions on matched pairs to maintain utility while enforcing invariance.