Inherent Unreliability of LLM-Generated Confidence Scores

LLMs lack a verifiable basis for generating trustworthy confidence scores because they do not possess an internal representation of the information they use to make predictions. Relying on LLM-generated probabilities for agentic decision-making is fundamentally flawed due to this lack of grounding.