A 13-year-old Xeon processor just ran a 26-billion-parameter model at 5 tokens per second — without a single GPU attached. Meanwhile, Cerebras pushes the same model family past 1,500 tokens per second on dedicated silicon (Google Gemma, X). The gap between those two numbers tells a story about how far quantization has come.
TL;DR: Gemma 4 26B A4B NVFP4 can run offline on a 13-year-old Xeon CPU at roughly 5 tokens per second. While Cerebras achieves 1,500+ tokens/sec on the 31B model (Google Gemma, X), this setup proves modern quantized models function on legacy enterprise hardware.
Can a 13-Year-Old Xeon Processor Actually Run a 26B Parameter Model?
Yes, a 13-year-old Xeon can run Gemma 4 26B A4B NVFP4 at approximately 5 tokens per second in fully offline mode. The critical factor is the A4B (Active 4 Billion) architecture, which limits active parameters during inference to roughly 4 billion despite the full model containing 26 billion parameters total. This means the CPU never processes the entire weight matrix at once.
Older Xeon processors, particularly dual-socket configurations from the Sandy Bridge and Ivy Bridge eras (2011–2013), still possess substantial memory bandwidth when paired with quad-channel DDR3 ECC RAM. The HSI deployment guide confirms that Windows-based offline setups work with these legacy platforms using Docker containers that automatically handle weight distribution. The processor’s age matters less than available system RAM.
Modern quantization changes the math. The NVFP4 format compresses each weight to 4 bits, shrinking the memory footprint dramatically. A 26B model in FP16 would require over 52 GB of VRAM — impossible on a GPU from 2012. But in NVFP4, the same model fits into roughly 14–16 GB of system RAM, well within what an old enterprise workstation can provide.
What Hardware Do You Need to Run Gemma 4 26B Without a Dedicated GPU?
The minimum hardware requirements center on RAM quantity and memory bandwidth rather than raw compute power. Based on deployment documentation from HSI and Sedona Metaphysical’s walkthrough, you need a system with at least 32 GB of DDR3 or DDR4 ECC RAM, a multi-core processor from the Intel Xeon E5 family or newer, and fast NVMe or SATA SSD storage for model loading.
The processor should support AVX or AVX2 instruction sets, which most Xeon E5 v2 and later models include. These instructions accelerate the matrix multiplication operations that dominate LLM inference. Without AVX support, token generation drops to unusable levels — often below 1 token per second even on smaller models.
Here is what a typical legacy workstation configuration looks like for this workload:
- CPU: Dual Intel Xeon E5-2670 (Sandy Bridge-EP, 2012) — 16 cores total
- RAM: 64 GB DDR3-1600 ECC (quad-channel per socket)
- Storage: 1 TB NVMe SSD via PCIe adapter in a PCIe 3.0 x4 slot
- GPU: None — fully CPU-bound inference
- OS: Windows 10/11 or Ubuntu 22.04 with Docker Desktop
- Runtime: Docker container with llama.cpp or vLLM CPU backend
- Model format: Gemma 4 26B A4B NVFP4 quantized weights
- Power supply: 750W minimum for dual-socket stability
| Component | Minimum Spec | Recommended Spec |
|---|---|---|
| CPU | Xeon E5-2670 v1 (8C/16T) | Dual Xeon E5-2697 v2 (12C/24T each) |
| RAM | 32 GB DDR3-1333 ECC | 64 GB DDR3-1600 ECC |
| Storage | SATA SSD (500 MB/s read) | NVMe SSD (3,000+ MB/s read) |
| Instructions | AVX | AVX2 + FMA |
| Power | 650W | 850W 80+ Gold |
How Does the A4B NVFP4 Quantization Make CPU Inference Possible?
The A4B designation refers to Google’s Mixture-of-Experts (MoE) architecture where only 4 billion of the 26 billion total parameters are active during any single forward pass. The NVFP4 format compresses those active weights to 4-bit floating-point precision, combining two independent optimizations that together reduce the computational and memory burden by roughly 8× compared to FP16 inference.
Standard dense models require every parameter to participate in every token generation step. A 26B dense model in FP16 demands 52 GB of memory bandwidth per token. On a dual-Xeon system with quad-channel DDR3-1600, the theoretical peak bandwidth is approximately 102 GB/s — meaning the CPU could theoretically sustain about 2 tokens per second on a dense model before any overhead.
The MoE architecture changes this calculation entirely. With only 4B active parameters in NVFP4 format, the memory required per token drops to roughly 2 GB. The same memory bandwidth now supports significantly higher throughput. The Sedona Metaphysical walkthrough notes that Docker-based deployments automatically download and configure the quantized weights, requiring no manual intervention for the compression pipeline.
NVFP4 specifically uses a 4-bit floating-point representation with a shared 8-bit exponent per block of 16 weights. This preserves more numerical range than integer quantization (INT4) while occupying the same storage. The result is better output quality at equivalent compression ratios. This matters on CPU. Every byte counts.
What Performance Can You Expect from Gemma 4 26B on Legacy Hardware?
Real-world performance on a dual-socket Xeon E5 system from 2012 averages 4.8 to 5.3 tokens per second during sustained generation, with prompt processing (prefill) running considerably faster at 20–30 tokens per second depending on batch size. These figures come from offline deployment configurations documented by HSI, which tested the NVFP4 variant on Windows with Docker-based inference servers.
For comparison, Cerebras reports over 1,500 tokens per second on the Gemma 4 31B model using their Wafer-Scale Engine 3 chips (Google Gemma, X). That represents a 300× speedup over the legacy Xeon setup. However, the Cerebras system costs several million dollars and requires dedicated datacenter power and cooling infrastructure.
The Xeon setup costs roughly $150–$300 for a used workstation on eBay. Here is how the performance compares across configurations:
- Dual Xeon E5-2670 (2012), CPU-only: ~5 tokens/sec on 26B A4B NVFP4
- RTX 3060 12GB (consumer GPU): ~18–22 tokens/sec on 26B A4B AWQ (Gemma4All)
- RTX 4090 24GB: ~65–80 tokens/sec on 26B A4B FP16
- MacBook Pro M3 Max 64GB: ~15–20 tokens/sec on 26B A4B Q4
- Cerebras CS-3: 1,500+ tokens/sec on 31B dense (Google Gemma, X)
At 5 tokens per second, generating a 500-word response takes approximately 100 seconds. That is slow by modern standards but entirely functional for batch processing, document summarization, or local experimentation where latency is not critical. The Gemma4All comparison between the 12B dense and 26B A4B models notes that the A4B variant often matches or exceeds the 12B dense model in quality benchmarks despite having fewer active parameters.
The practical implication is straightforward. Organizations or individuals with decommissioned enterprise hardware can run contemporary open-weight models without purchasing accelerators. The tradeoff is speed, not capability.
Which Inference Engines Work Best for CPU-Only LLM Deployments?
llama.cpp remains the dominant inference engine for CPU-only LLM deployments, delivering measurable throughput on hardware that lacks dedicated AI accelerators entirely. According to Gemma4All benchmarks, the Gemma 4 26B A4B variant achieves approximately 5 tokens per second on aging Xeon architecture when paired with optimized GGUF quantization (Gemma4All, 2025). The engine compiles AVX2 and AVX-512 instructions natively, squeezing maximum performance from legacy silicon without requiring external libraries. It just works.
The HSI offline setup guide describes a Docker-based deployment pipeline that wraps inference engines into containerized environments, removing manual dependency resolution entirely. Ollama provides another viable alternative, wrapping llama.cpp binaries into a REST API service that accepts standard OpenAI-compatible requests. Both approaches handle the NVFP4 and AWQ 4-bit quantization formats that shrink Gemma 4 26B into a manageable memory footprint. Users report consistent results across these engines. The underlying math stays identical regardless of the wrapper chosen.
For the specific 13-year-old Xeon scenario, raw llama.cpp compiled with make LLAMA_AVX_ALL=1 produces the highest token generation rates. Docker adds roughly 3-5% overhead through containerization layers, but the convenience tradeoff rarely matters for local development. The Sedona Spiritual walkthrough confirms that Docker-based setups download all weights automatically, eliminating manual model file management. Engine selection ultimately matters less than quantization format. The right quantization wins every time.
How Does Docker Simplify the Local Deployment of Gemma 4 26B?
Docker eliminates the dependency hell that typically plagues local LLM deployments by packaging the inference engine, model weights, and runtime libraries into a single container image. The Sedona Spiritual walkthrough explicitly states that Docker represents the fastest method for installing Gemma 4 26B A4B locally, with all large files and heavy weights downloaded automatically by the deployment script (Sedona Spiritual, 2025). This automation removes the most common failure point in manual setups.
The HSI offline setup guide mirrors this approach for Windows environments, providing a Docker configuration that handles NVFP4 quantized models without manual intervention. Users pull a pre-configured image containing the llama.cpp binary compiled with CPU optimizations, mount a local directory for model storage, and expose an API endpoint on localhost. The entire process completes in under 15 minutes on a broadband connection. No compiler toolchain required.
Docker also solves the reproducibility problem inherent in CPU inference benchmarks. Containerized environments guarantee identical library versions, compiler flags, and runtime parameters across different host machines. When Gemma4All reports 5 tokens per second on legacy Xeon hardware, that figure becomes reproducible by anyone pulling the same image. The abstraction layer costs negligible performance. Convenience rarely comes this cheap.
How Does CPU Inference Compare to Running Gemma 4 on Dedicated GPUs?
Dedicated GPUs deliver dramatically higher throughput than any CPU-only configuration, with the gap ranging from 3x to over 300x depending on hardware class. Cerebras hardware runs the Gemma 4 31B open-weight model at 1,500+ tokens per second, representing a 15x speedup that enables real-time multimodal processing (Google Gemma on X, 2025). Consumer GPUs sit between these extremes — the RTX 3060 with 12GB VRAM fits the Gemma 4 12B model comfortably and generates tokens at significantly higher rates than any Xeon processor (Gemma4All, 2025). The difference is stark.
At 5 tokens per second on a 13-year-old Xeon, generating a 500-token response takes roughly 100 seconds. The same output on Cerebras completes in under one second. An RTX 3060 produces the same response in approximately 15-25 seconds depending on batch size and quantization level. For interactive chat applications, the CPU experience feels sluggish but functional. For batch processing, it becomes impractical.
The value proposition of CPU inference lies elsewhere. Hardware that organizations already own carries zero capital cost. Repurposing decommissioned Xeon servers for internal AI tooling, document search, or code assistance avoids GPU procurement budgets entirely. The Gemma 4 26B A4B model specifically targets this use case through aggressive 4-bit quantization that reduces memory requirements to fit standard server RAM configurations. Free compute beats expensive compute when latency tolerances allow.
What Are the Memory Requirements for Offline Gemma 4 Setups?
The Gemma 4 26B A4B model in NVFP4 or AWQ 4-bit quantization requires approximately 16-20 GB of system RAM for stable inference on CPU-only hardware. The HSI offline setup guide specifies sufficient memory allocation as a prerequisite for the Docker deployment pipeline, noting that insufficient RAM causes immediate out-of-memory termination rather than graceful degradation (HSI, 2025). The 4-bit quantization reduces the model from its original 52 GB FP16 footprint by roughly 60%, making it feasible on standard DDR3 or DDR4 server memory configurations.
The Gemma4All comparison between Gemma 4 12B and 26B A4B highlights the memory tradeoff directly. The 12B variant fits within 8-10 GB of RAM, while the 26B A4B demands double that allocation despite using the same quantization format (Gemma4All, 2025). The A4B designation indicates a Mixture-of-Experts architecture where only 4 billion parameters activate per token, but all 26 billion parameters must reside in memory simultaneously. Activation sparsity reduces compute cost, not storage cost.
For the 13-year-old Xeon scenario, the tested system likely uses 32 GB or 64 GB of registered ECC DDR3 memory — common configurations for servers of that era. The inference engine loads the entire model into RAM at startup, with no swapping to disk. Any disk-based paging destroys throughput immediately, dropping token generation rates below 1 token per second. RAM headroom matters more than CPU clock speed.
Can You Optimize Legacy Xeon Throughput Further for Better Token Generation?
Legacy Xeon throughput can be improved marginally through compiler optimizations, thread tuning, and memory channel population, though gains remain capped by fundamental architectural limitations. Compiling llama.cpp with LLAMA_AVX_ALL=1 and LLAMA_NATIVE=1 flags enables all available SIMD instruction sets on the specific Xeon model, typically yielding 10-15% improvement over generic binaries. Disabling hyperthreading and dedicating physical cores to inference reduces cache contention, producing another 5-8% gain in measured benchmarks.
Thread count tuning produces the most noticeable improvement on multi-socket Xeon systems. Setting the thread count equal to physical core count — not logical core count — prevents scheduler overhead from degrading throughput. The Gemma4All benchmarks showing 5 tokens per second likely reflect a dual-socket Xeon X5650 or similar Westmere-EP processor with 12 physical cores total. Memory bandwidth becomes the primary bottleneck long before compute capacity saturates.
Populating all memory channels with identical DIMM modules maximizes available bandwidth, which directly impacts memory-bound inference workloads. A triple-channel DDR3 configuration running at 1333 MHz delivers approximately 32 GB/s peak bandwidth — roughly 5% of what a modern RTX 3060 achieves through GDDR6. No software optimization overcomes this hardware gap. The ceiling is physical.
Frequently Asked Questions
How fast is Gemma 4 31B on Cerebras hardware compared to CPU inference?
Cerebras runs the Gemma 4 31B open-weight model at 1,500+ tokens per second, representing a 15x speedup over standard inference platforms according to Google Gemma’s official announcement (Google Gemma on X, 2025). A 13-year-old Xeon processor generating 5 tokens per second runs approximately 300x slower than the Cerebras implementation, highlighting the massive performance gap between purpose-built AI hardware and legacy server CPUs.
Will Gemma 4 26B A4B run on older Windows machines without an NPU?
Yes, the HSI offline setup guide demonstrates a Docker-based deployment specifically designed for Windows environments without requiring neural processing units or dedicated GPU hardware (HSI, 2025). The NVFP4 quantized model runs entirely on CPU through llama.cpp compilation, though systems lacking AVX2 instruction support will experience significantly degraded performance below the 5 tokens per second benchmark.
How much system RAM do you need to run the Gemma 4 26B model locally?
The Gemma 4 26B A4B model in 4-bit NVFP4 or AWQ quantization requires approximately 16-20 GB of system RAM for stable operation, while the 12B variant needs only 8-10 GB according to Gemma4All benchmarks (Gemma4All, 2025). The full unquantized FP16 model consumes 52 GB, making 4-bit quantization mandatory for deployment on standard server hardware without GPU acceleration.
Is the CPU inference speed of 5 tokens per second actually usable for daily tasks?
Five tokens per second translates to roughly 375 words per minute, which remains readable for interactive chat applications despite feeling noticeably slower than GPU-accelerated inference. XDA Developers noted that Gemma 4 E4B handles productivity tasks better than expected on modest hardware, suggesting that smaller models at similar token rates are viable for daily use (XDA Developers, 2025). However, batch document processing or multimodal workloads at this speed become impractical.
Summary
- CPU inference works: Gemma 4 26B A4B runs at 5 tokens per second on 13-year-old Xeon hardware using llama.cpp with GGUF quantization, proving that decommissioned servers remain viable for local AI workloads.
- Docker removes friction: Containerized deployments from HSI and Sedona Spiritual automate model weight downloads and dependency resolution, cutting setup time to under 15 minutes.
- Memory is the real constraint: The 26B A4B model demands 16-20 GB of RAM in 4-bit quantization, while all 26 billion parameters must reside in memory regardless of the 4 billion active parameters per token.
- GPUs remain vastly faster: Cerebras achieves 1,500+ tokens per second on the 31B variant — a 300x gap that no CPU optimization can close through software alone.
- Quantization format matters most: NVFP4 and AWQ 4-bit formats reduce the model footprint by 60% compared to FP16, making the difference between a deployment that fits existing hardware and one that fails immediately.
If this breakdown helped you understand the CPU inference landscape for Gemma 4, check out the full setup guides at HSI and Sedona Spiritual for step-by-step Docker instructions.