Research shows that reported performance gaps in structured prediction are joint properties of the model and the evaluation contract, including scoring functions and prompt language. The study introduces the Audited Evaluation Reading (AER) to mitigate contract-induced rank flips.
HOW THIS AFFECTS YOU
●
researcherYou must account for the specific evaluation contract when comparing model capability improvements.