AirLLM enables 70B model inference on 4GB GPUs via expert streaming
August 3, 2026
AirLLM achieves low-memory inference for large models by streaming individual experts for Mixture-of-Experts architectures instead of loading entire layers. This allows 70B models to run on 4GB GPUs and the 2.8T Kimi K3 model to run on 3.72GB VRAM on a single RTX 6000 Ada.
HOW THIS AFFECTS YOU
●
builderYou can deploy massive MoE models on consumer-grade or low-VRAM hardware without resorting to quantization.
●
founderThis lowers the hardware barrier for deploying high-parameter model capabilities in edge or cost-constrained environments.