CleanRM: Calibrating Sample-Wise Noise in Reward Modeling
October 16, 2026
CleanRM uses a small amount of clean feedback and optimal transport to calibrate sample-wise noise ratios in preference data. This approach corrects for non-uniform noise from human or LLM annotators, providing an unbiased estimate of clean risk during RLHF.
HOW THIS AFFECTS YOU
●
builderYou can improve RLHF stability if your preference datasets contain inconsistent or noisy labels.
●
researcherThis provides a theoretically unbiased objective for reward modeling under heterogeneous noise distributions.