How much information do LLM weights actually carry?

New research by fergusfinn examines information-theoretic limits of LLM weight compression, analyzing entropy bounds that constrain how aggressively weights can be quantized before irreversible capacity loss. The work frames inference as a memory-bandwidth versus compute tradeoff, directly relevant to practitioners deploying quantized models like GPTQ or AWQ at 4-bit and below. Engineers optimizing edge deployments should pay close attention.