Stated vs. Revealed Preferences Diverge in LLM Human Feedback
October 23, 2026
Analysis of the PRISM dataset reveals that users' stated preferences often mismatch their actual ratings of LLM responses. Unsurveyed attributes like response length, formatting, and refusal language significantly influence human rewards, sometimes more than the specific attributes users claim to prioritize.
HOW THIS AFFECTS YOU
●
builderWhen designing agent personas, realize that users may favor formatting and length over core utility.
●
researcherConsider that RLHF objectives may inadvertently optimize for surface-level features rather than stated user intent.