A 28.9M parameter model achieves 9.5 tokens per second on an $8 ESP32-S3 microcontroller by utilizing per-layer embeddings to store 25M parameters in flash memory rather than SRAM. This approach increases model capacity by 100x compared to previous microcontroller implementations while maintaining local, offline execution.
HOW THIS AFFECTS YOU
●
builderYou can deploy larger language models on low-cost, low-power edge hardware using flash-resident embeddings.
●
founderThis lowers the hardware barrier for creating offline, privacy-first AI consumer products at scale.