An $8 microcontroller now runs a 28.9-million-parameter language model entirely offline. Developer slvDev achieved this on the ESP32-S3, generating text at roughly 9.5 tokens per second without any cloud connection or external server. The chip costs less than a fast-food lunch.
TL;DR: A developer successfully ran a 28.9-million-parameter language model completely offline on an $8 ESP32-S3 microcontroller. Using 4-bit quantization and per-layer flash storage, the chip achieves 9.5 tokens per second without needing cloud infrastructure.
How Does a 28.9M Parameter LLM Fit on an $8 ESP32-S3?
The ESP32-S3 packs 512KB of SRAM — a fraction of what a 28.9M-parameter model needs for storage. The breakthrough relies on storing model weights in the chip’s flash memory rather than RAM, using a technique called per-layer embedding. Combined with 4-bit quantization, the entire model occupies roughly 17MB of flash space, which the chip reads sequentially during inference. This approach grew out of Andrej Karpathy’s llama2.c project and Google’s research into per-layer embeddings, which allows large models to live in flash instead of RAM. The result is a model roughly 100 times larger than what people ran on the same chip in 2023. That is massive progress in under three years.
The core problem is bandwidth. To generate each token, the processor must read the entire model from memory. When a model lives in fast SRAM, this read operation happens quickly. Flash memory is slower, but the ESP32-S3 can stream data from flash at sufficient speed to make 9.5 tokens per second achievable. The 4-bit quantization halves the data the chip must move per parameter, trading a small accuracy reduction for a dramatic bandwidth improvement. This tradeoff makes on-chip inference viable.
What Is Per-Layer Embedding Storage in Flash Memory?
Per-layer embedding storage is the technique that makes this entire project possible. Instead of loading the full model into the ESP32-S3’s limited 512KB SRAM, the system stores all 28.9 million parameters in the chip’s external flash memory. The processor reads each layer’s weights from flash on demand during the forward pass, processes them, and discards them before moving to the next layer. This streaming approach means the microcontroller never needs to hold the entire model in working memory at once. Only the active layer matters at any given moment.
Traditional model inference loads everything into RAM for speed. That works on devices with gigabytes of memory, but a microcontroller with 512KB cannot follow that playbook. SlvDev’s implementation treats flash storage as a sequential read buffer. The tradeoff is latency — flash reads are slower than SRAM accesses. However, because the model uses 4-bit quantization, each parameter requires only half a byte. That means the chip reads roughly 14.5MB of data per generated token. The ESP32-S3’s flash interface handles this throughput well enough to sustain nearly 10 tokens per second. The math works out because the bandwidth requirements stay within the hardware’s limits.
For comparison, in 2024 Dave Bennett loaded a 260,000-parameter model into PSRAM and achieved 19 tokens per second on similar hardware. That model was tiny — roughly 1MB total — so reading all of it per token was computationally cheap. The per-layer flash approach scales better because it decouples model size from available RAM. Make the model bigger and each token requires more flash reads, but the system never runs out of memory. That architectural difference is what enables a 28.9M-parameter model to coexist with a 512KB memory budget.
How Fast Does the ESP32-S3 Generate Text Locally?
The ESP32-S3 generates text at approximately 9.5 tokens per second when running the 28.9M-parameter model. That speed puts it in a usable range for interactive applications — slow enough to watch the text appear, but fast enough to feel responsive on a connected display. For context, many cloud-based LLM APIs generate at 20 to 60 tokens per second, meaning the $8 chip runs at roughly one-third to one-half the speed of hosted inference. The difference is that no network connection is required. Everything happens on the silicon.
The generation speed depends on several factors. The 4-bit quantization reduces data movement, but the flash read bandwidth remains the primary bottleneck. Each generated token requires a full pass through all 28.9 million parameters. At 4 bits per parameter, that translates to reading about 14.5 megabytes from flash for every single output token. The ESP32-S3’s SPI flash interface sustains enough throughput to complete this read in approximately 100 milliseconds, yielding the observed 9.5 tokens per second. Clock speed and cache efficiency also play a role. The chip’s dual-core processor handles the matrix arithmetic while flash reads happen in parallel.
SlvDev demonstrated the system writing a short story directly to a screen wired to the chip. The text appeared progressively, similar to watching a typewriter output. No data left the device. No server processed the request. The entire inference pipeline — tokenization, embedding lookup, transformer layers, and decoding — executed on the microcontroller itself. That self-contained operation is what distinguishes this from edge AI approaches that offload computation to a phone or local PC.
What Hardware Do You Need to Run This Microcontroller LLM?
The hardware requirements are minimal by design. The core component is an ESP32-S3 module with sufficient flash storage — at least 16MB to hold the quantized model weights and the inference firmware. The specific board slvDev used costs approximately $8 from standard electronics suppliers. Beyond the chip itself, the setup includes a display for output and basic wiring to connect the screen to the ESP32-S3’s GPIO pins. No GPU is needed. No neural processing unit is required. The ESP32-S3’s dual-core CPU handles everything.
Here is what a typical build requires:
- ESP32-S3 development board — N8R2 or N16R8 variant with at least 8MB flash
- SPI display — ST7789 or ILI9341 based, 240x240 or 320x240 resolution
- Breadboard and jumper wires — for connecting display to GPIO pins
- USB-C cable — for flashing firmware and power supply
- Computer with PlatformIO or Arduino IDE — for compiling and uploading the project
- 3.3V power source — USB power works, battery optional for portable use
- Optional: battery pack — for fully untethered offline operation
- Optional: tactile buttons — for triggering text generation or resetting output
| Component | Specification | Approximate Cost |
|---|---|---|
| ESP32-S3 board | N16R8, 16MB flash, 8MB PSRAM | $7–$9 |
| SPI TFT display | 1.3” or 1.54”, ST7789 driver | $3–$5 |
| Breadboard + wires | Standard 830-point breadboard | $2–$3 |
| USB-C cable | Data + power, 1m length | $2–$4 |
| Total | Complete offline AI device | $14–$21 |
The total bill of materials stays under $21 even with a display and accessories. The bare ESP32-S3 module alone costs $8. For developers who already have microcontroller components in their parts bin, the incremental cost to replicate this project could be close to zero. The flash storage capacity is the most critical specification — the model needs space to live. Boards with only 4MB of flash cannot hold the full 28.9M-parameter model at 4-bit precision. The 16MB variant provides comfortable headroom for the weights, firmware, and a small vocabulary buffer.
How Does This Compare to Earlier ESP32 AI Models?
Running a 28.9M-parameter model on an ESP32-S3 represents roughly a 100x scale-up from what was possible on the same chip in 2023, according to developer Slava S. (slvDev). Earlier efforts focused on much smaller models that fit entirely in PSRAM. Dave Bennett’s 2024 project, for instance, loaded a 260K-parameter model (~1MB) into PSRAM and achieved 19 tokens per second. That worked because reading the entire model for each token was computationally cheap at that size.
The problem is that approach does not scale. As the model grows, every single token generation requires reading more bytes from memory. The ESP32-S3 has only 512KB of SRAM, which is nowhere near enough for a 28.9M-parameter model. Slava S. solved this by storing most parameters in flash memory rather than RAM, using a technique called Gemma Per-Layer Embeddings. This allows a large model to live in flash and be read layer by layer during inference.
So the tradeoff is clear. The 260K model ran faster at 19 tok/s, but the 28.9M model produces far more coherent output at roughly 9.5 tok/s. The new approach sacrifices some speed for a massive increase in model capacity.
For context, the very first version of ChatGPT had 117 million parameters, as noted by Brian Roemmele. That means this $8 chip runs roughly a quarter of the parameters of OpenAI’s debut model — entirely offline, with no server round-trips.
What Is the Software Stack Behind This Offline LLM?
The software foundation traces back to two key projects. The first is Andrej Karpathy’s llama2.c, a minimalist C implementation of LLaMA-2 inference designed to run on resource-constrained hardware. The second is Google’s Per-Layer Embeddings research, which enables large models to reside in flash memory instead of requiring fast RAM access for the entire parameter set.
The ESP32-S3 chip itself provides the hardware foundation. It features a dual-core Xtensa LX7 processor, 512KB of SRAM, and support for external flash storage. The model is quantized to 4-bit precision, which dramatically reduces the storage footprint while maintaining acceptable output quality. At 4-bit quantization, 28.9 million parameters fit into a manageable flash footprint that the chip can read sequentially.
The inference loop works as follows. For each token generated, the chip reads the entire model from flash memory, layer by layer. This is inherently slower than reading from RAM, but flash storage on the ESP32-S3 is large enough to hold the full model. The per-layer embedding technique means only one layer’s worth of parameters needs active processing at any given moment.
The project outputs text directly to a screen wired to the chip. No external processor, no cloud API, no network stack required for inference. The entire pipeline runs on the microcontroller.
Can This Microcontroller AI Replace Cloud Inference for IoT?
This project challenges the assumption that AI inference requires cloud connectivity. For IoT devices, cloud-based AI introduces latency, privacy concerns, recurring API costs, and dependency on network availability. An $8 ESP32-S3 running a local LLM eliminates all four of those constraints.
Brian Roemmele described the project as opening “a massive opportunity for dozens of these devices” in distributed deployments. Consider a sensor network in a remote location without reliable internet. Each node could perform basic language reasoning locally — summarizing sensor readings, generating status reports, or responding to simple queries — without phoning home to a server.
However, the 28.9M-parameter model has limited reasoning capacity compared to cloud models with billions of parameters. It generates coherent short stories at approximately 9.5 tokens per second, but it cannot match the output quality of GPT-4 or Claude. The use case is narrow: simple text generation tasks where offline operation matters more than sophisticated reasoning.
For applications like smart home controllers, industrial sensor hubs, or offline educational devices, this level of local AI could be sufficient. The economics are compelling — a one-time $8 hardware cost versus ongoing cloud API charges.
What Are the Limitations of Running LLMs on Microcontrollers?
The most significant limitation is memory bandwidth. As Slava S. explained in his technical breakdown, generating each token requires reading the entire model from flash. Flash memory on the ESP32-S3 is slow compared to RAM, which caps generation speed at roughly 9.5 tokens per second for this model size.
The chip’s 512KB of SRAM places a hard constraint on what can be held in fast memory simultaneously. Even with per-layer embeddings and 4-bit quantization, the architecture cannot scale to arbitrarily large models. Pushing beyond 28.9M parameters on this specific chip would require either aggressive pruning, a different quantization strategy, or a chip with more SRAM.
Model quality is another constraint. At 28.9M parameters, the model produces coherent short text but lacks the depth, factual accuracy, and instruction-following capability of larger models. The first version of ChatGPT had 117M parameters and was already limited in many ways — this model has roughly one-quarter of that capacity.
There are also practical hardware constraints. The current setup writes text to a directly attached screen. There is no networking stack, no audio input, and no multimodal capability in the base configuration. Integrating the LLM with sensors, actuators, or communication interfaces would require additional engineering work within the chip’s limited I/O and processing budget.
Power consumption, while low for a microcontroller, still requires a stable power source for continuous inference. Battery-powered deployments would need careful power management.
How Do You Set Up the ESP32-S3 for Local AI Inference?
Setting up the ESP32-S3 for local LLM inference requires specific hardware and software components. The hardware list starts with an ESP32-S3 development board, which typically costs around $8. You also need a compatible display module (such as an SPI or I2C screen) to view the generated text output, plus wiring and a USB cable for flashing firmware.
The software setup involves several steps. First, install the ESP-IDF (Espressif IoT Development Framework) or PlatformIO toolchain for building and flashing firmware to the ESP32-S3. Clone the project repository, which is built on top of Karpathy’s llama2.c adapted for the ESP32 platform. The model weights, quantized to 4-bit precision, must be compiled and flashed to the chip’s external flash storage.
Here is a simplified overview of the build process:
# Clone the repository
git clone <project-repo>
cd esp32-llm
# Set up ESP-IDF environment
. $HOME/esp/esp-idf/export.sh
# Configure the project for ESP32-S3
idf.py set-target esp32s3
# Build and flash
idf.py build
idf.py -p /dev/ttyUSB0 flash monitorThe model file must be converted to a format compatible with the per-layer embedding approach. This involves preprocessing the original weights so each transformer layer can be loaded from flash sequentially during inference. The project documentation provides conversion scripts for this purpose.
Key setup considerations include:
- Board selection: Not all ESP32-S3 boards have identical flash sizes; verify you have sufficient flash (typically 8MB or more)
- Partition table: The default partition scheme may need adjustment to allocate space for the model file
- Display wiring: Pin assignments for the SPI display must match the firmware configuration
- Power supply: Stable 3.3V power is essential; brownouts during flash writes can corrupt the model
- Clock speed: The ESP32-S3 can run at 240MHz; ensure the firmware configures the CPU at maximum frequency for best inference speed
- Flash mode: QIO (Quad I/O) flash mode provides faster reads than standard mode
- PSRAM: While the model lives in flash, having PSRAM available helps with intermediate computations
- Monitoring: Connect via serial monitor at 115200 baud to observe inference logs and debug issues
The table below summarizes the key hardware and software requirements:
| Component | Requirement | Notes |
|---|---|---|
| Microcontroller | ESP32-S3 | 512KB SRAM, dual-core 240MHz |
| Flash storage | 8MB+ external flash | Holds the 4-bit quantized model |
| Display | SPI TFT (e.g., ILI9341) | Shows generated text output |
| Framework | ESP-IDF or PlatformIO | Build and flash toolchain |
| Model format | 4-bit quantized, per-layer | Converted from standard weights |
| Base code | llama2.c (Karpathy) | Adapted for ESP32 architecture |
Once flashed, the chip boots directly into inference mode. No network configuration, API keys, or cloud credentials are needed. The model loads from flash and begins generating text to the attached display.
Frequently Asked Questions
Can the ESP32-S3 run models larger than 28.9M parameters?
The current implementation caps at 28.9M parameters due to the 512KB SRAM limit and flash read bandwidth on the ESP32-S3. The developer, Slava S., noted that the approach grew out of Karpathy’s llama2.c and Google’s per-layer embeddings, which is what allows a big model to live in flash instead of RAM. Pushing significantly beyond this size would likely require a chip with more SRAM or a fundamentally different memory architecture.
Does the ESP32-S3 LLM require an internet connection to generate text?
No. The model runs completely offline on the chip with nothing sent to any server, as confirmed by Slava S. in his demonstration. The ESP32-S3 stores all 28.9 million parameters in its own flash memory and performs inference locally, writing text directly to an attached screen at approximately 9.5 tokens per second. This makes it suitable for deployments where network connectivity is unreliable or unavailable.
How does the token generation speed compare to cloud-based LLMs?
The ESP32-S3 generates text at roughly 9.5 tokens per second, which is slower than most cloud-hosted LLM APIs. For comparison, Dave Bennett’s 2024 project achieved 19 tokens per second with a much smaller 260K-parameter model loaded into PSRAM, because reading that ~1MB model per token was cheap. The 28.9M model trades speed for capacity — the chip must read the full model from flash for every single token generated.
Is the 28.9M parameter model useful for real-world applications?
The model generates coherent short stories and basic text, but it has roughly one-quarter the parameter count of the original ChatGPT, which had 117 million parameters according to Brian Roemmele. It is not capable of complex reasoning, factual question-answering, or sophisticated instruction following. However, for simple offline text generation tasks on IoT devices, sensor hubs, or educational tools, the capacity is sufficient — and the $8 hardware cost with zero ongoing API charges makes it economically viable for specific edge AI use cases.
Summary
The ESP32-S3 LLM project demonstrates several key principles for edge AI developers:
- Flash-based inference works: Per-layer embeddings enable models far larger than available SRAM by reading parameters from flash sequentially, achieving a ~100x scale-up over 2023 efforts on the same chip
- Offline AI is economically viable: An $8 one-time hardware cost replaces recurring cloud API fees, making distributed AI deployments feasible at scale
- Speed-capacity tradeoffs are fundamental: Faster inference requires faster memory; the 28.9M model runs at ~9.5 tok/s from flash versus 19 tok/s for a 260K model from PSRAM
- Small models have real use cases: While no match for billion-parameter cloud models, a 28.9M-parameter LLM can serve IoT devices, offline educational tools, and privacy-sensitive applications where cloud connectivity is unnecessary or undesirable
As microcontroller hardware improves and quantization techniques advance, the boundary between edge AI and cloud AI will continue to shift. This project from slvDev, built on Karpathy’s llama2.c and Google’s per-layer embeddings research, provides a practical blueprint for anyone exploring local language model inference on ultra-low-cost hardware.