LatentRM Learns Intermediate Reasoning Traces for Scalar Reward Models

LatentRM addresses the gap between scalar reward models and generative reasoning models by learning intermediate reasoning traces as discrete latent variables. This approach forces reasoning paths to explicitly maximize the likelihood of downstream scalar reward predictions, improving generalization on out-of-distribution tasks.