Hybrid VRAM and system RAM speculative decoding for large models
October 21, 2026
A proposed optimization method uses a 2-bit model copy in GPU VRAM for drafting tokens while keeping 6-8 bit residual weights in system RAM. The CPU handles the residual part to minimize PCIe bandwidth usage, using the full model as a verifier.
HOW THIS AFFECTS YOU
●
builderYou may be able to run larger models on limited hardware by offloading quantized weights to system RAM.
●
researcherThis approach presents a new way to optimize speculative decoding using heterogeneous memory hierarchies.