Bayesian Preference Learning for Test-Time Steerable Reward Models

Variational In-Context Reward Modeling (ICRM) enables test-time steerability by treating reward modeling as amortized variational inference using a Beta prior under the Bradley-Terry model. This method adapts to new preference distributions via in-context demonstrations, increasing RM-Bench accuracy from 60.5 to 70.8 in single and multi-objective settings.