Predicting Medical Item Difficulty via LLM Response Signals

Researchers utilize open-weight LLMs as both judges and examinees to predict medical exam difficulty for USMLE and PANCE items. The method combines pairwise difficulty judgments with response-based signals like response entropy and keyed-option probability to improve predictive performance.