Internal Log-Probabilities Predict Zero-Shot LLM Annotation Performance

Internal log-probabilities are more predictive of an LLM's success in zero-shot text classification than verbalized self-confidence scores. Testing across 461 datasets shows that these internal signals can be used to estimate how well a model will annotate a new, unseen dataset.