Multi-Stage Architecture for Learning Heterogeneous Human Preferences

A new framework moves beyond the assumption of universal utility by introducing individuated utility functions that condition on both the person and their context. The multi-stage architecture estimates these heterogeneous preferences from multi-modal data to improve RLHF in subjective domains.