Two-Stage LLM Cascade Retains 97–99% Accuracy While Cutting Costs

A cascaded serving framework first clusters queries and routes each cluster to the cheapest sufficient model, then applies a quality estimation layer to escalate low-confidence outputs to stronger models. On test datasets, the system retains 97–99% of the strongest model's accuracy with significantly reduced cost, controlled by a single offline-tunable hyperparameter.