Predicting LLM Performance on Unseen Questions via Multidimensional IRT

This framework uses multidimensional item response theory and question embeddings to predict model performance on new tasks without new annotations. By representing models through latent capability profiles, the method allows information to transfer from observed items to unobserved questions within the same scenario.