Quantifying Option Bias via Flip Rate and Positional Skew
October 16, 2026
A new framework decomposes multiple-choice evaluation bias into flip rate, measuring answer changes across orderings, and positional skew, measuring label-letter preference. Testing across 41 checkpoints shows flip rate correlates with model accuracy and uncertainty.
HOW THIS AFFECTS YOU
●
researcherYou can use these two factors to more accurately diagnose whether your model's benchmark performance is driven by actual knowledge or positional heuristics.