Prompt Format Significantly Influences AI Preference Measurements

An analysis of 11,400 elicitations across five different prompt formats shows that the choice of instrument heavily dictates measured model preferences. Holding outcomes and models fixed reveals that preference measurement is highly sensitive to the specific prompting methodology used.