LLM Non-Determinism Found in Medical Device Document Assessment

Testing fourteen commercial LLM deployments revealed that even with deterministic sampling, models frequently return inconsistent assessments for medical regulatory compliance. Variation persisted despite reducing reasoning steps, and reproducibility often depended on the specific serving infrastructure rather than the model weights.