OTROPE enables likelihood-free off-policy evaluation for black-box LLMs
September 30, 2026
OTROPE uses optimal transport to perform distributional correction in semantic space, allowing for robust off-policy evaluation of LLMs. It aligns labeled behavior-policy samples with unlabeled target-policy samples without requiring behavior-policy modeling or density-ratio estimation.
HOW THIS AFFECTS YOU
●
builderYou can evaluate new models using existing human-labeled datasets even when working with black-box APIs.
●
researcherThis provides a new method for evaluating target models when response likelihoods are unavailable.