Decentralized P2P Prefix-Cache Routing Cuts LLM Inference Latency Without Central Coordination

A decentralized routing scheme uses per-node radix trees and async anti-entropy gossip to match incoming requests to the peer with the longest cached prefix, avoiding KV-cache transfers and centralized coordinators. On simulated MMLU workloads, the approach reduces latency under low network delay and skewed prefix distributions, though high-latency networks and hotspots limit gains. Stale metadata causes only cache misses, not correctness failures.