GraphTok implements the entire BPE encoding pipeline as a replayable CUDA graph, moving tokenization from the CPU to the GPU. It achieves 4.7-6.0x faster performance than existing CPU-based tokenizers on Llama 3 and integrates with vLLM and NVIDIA Dynamo.
HOW THIS AFFECTS YOU
●
builderYou can drastically reduce time-to-first-token (TTFT) for agentic workloads by offloading tokenization to the GPU.