High-Performance Rate Limiting at Databricks

Databricks redesigned its rate limiter for ML model serving, replacing synchronous Redis checks with asynchronous batch reporting using in-memory sharded counters and token bucket algorithms. The system handles millions of RPS while accepting ~5% overshoot to slash latency. Critical reading for ML platform engineers building high-throughput inference infrastructure.