Limits of Verbalized Uncertainty and Calibration in Small Language Models

Evaluation of eleven instruction-tuned models (0.5B to 14B parameters) shows that temperature scaling fails to calibrate models when confidence remains above 0.5 while accuracy drops below it. Researchers can use a Clopper-Pearson procedure to convert calibration sets into finite-sample risk certificates for offline deployment.