FreeToken lets a single GPU run 753B GLM-5.2 locally

A team of researchers from the University of California, Berkeley and the University of Texas at Austin announced FreeToken, an Apache‑2.0 licensed serving system that treats a personal computer as a flexible inference platform. By dynamically allocating work across the GPU, CPU, system memory and PCIe bandwidth, FreeToken can host the full parameter set of very large mixture‑of‑experts (MoE) models while only activating the experts needed for each token. The project is available on GitHub, packaged on PyPI as freetoken v0.1.2, and distributed as a one‑click desktop application for Windows and Linux.

The core innovation lies in how FreeToken handles cache misses for MoE experts. When an expert is not resident on the GPU, the engine decides whether to fetch it over PCIe or to execute it on the CPU, based on measured bandwidth and latency. This approach lets a modest 8 GB laptop GPU run a 35‑billion‑parameter model at interactive speeds, a 284‑billion‑parameter DeepSeek‑V4‑Flash model on a typical gaming desktop, and even the 753‑billion‑parameter GLM‑5.2 model on a single workstation‑class RTX PRO 6000 card. Performance measurements show 14.9 tokens per second for GLM‑5.2, compared with 7.3 t/s for the best open‑source baseline.

The release arrives at a time when open‑weight frontier models such as Kimi‑K3, GLM‑5.2 and DeepSeek‑V4‑Flash are narrowing the gap with proprietary systems. However, the cost of running these models has remained tied to datacenter‑grade GPU clusters, limiting access for solo developers, startups, and small research groups. More than a hundred million consumer PCs already contain discrete GPUs, and FreeToken aims to turn that latent hardware into a viable inference engine for high‑capacity models.

In head‑to‑head tests against popular open‑source engines, including llama.cpp, KTransformers, Ollama and MoE‑Infinity, FreeToken consistently outperformed the competition. On an RTX 5090, it sustained 77–83 tokens per second on Qwen3.6‑35B‑A3B (BF16) and 22–25 t/s on DeepSeek‑V4‑Flash (MXFP4), representing a 1.5‑ to 2.3‑fold speedup over the strongest baseline. First‑token latency stayed below 44 seconds across three agentic workloads, whereas llama.cpp, Ollama and KTransformers recorded worst‑case latencies of 232 s, 179 s and 946 s respectively. The engine also reduced cache miss rates: with equal cache capacity, its global LRU missed only 16 % of expert reads during decoding, versus 41 % for KTransformers and 62 % for llama.cpp.

Despite the impressive gains, FreeToken is not a universal replacement for large‑scale clusters. The system still depends on a capable GPU and sufficient PCIe bandwidth; inactive experts occupy roughly 140 GB of host memory at FP4 precision, which may strain machines with limited RAM. The authors caution that enterprises should view FreeToken as an air‑gapped or regulated‑workload path rather than a full‑scale production alternative. Its strongest use cases are in sectors where data cannot leave the device, such as healthcare, legal, defense, finance and intellectual‑property‑heavy research.

Looking ahead, the open‑source nature of FreeToken invites community contributions to improve expert scheduling, memory management and multi‑node scaling. The developers have already provided a command‑line interface that exposes OpenAI‑compatible and Anthropic‑compatible endpoints, enabling local coding assistants, private code review tools, offline contract analysis and synthetic data generation. As more developers experiment with edge‑native inference, the ecosystem may see a shift toward privacy‑preserving AI services that run entirely on user hardware.

FreeToken demonstrates that the bottleneck for frontier model deployment is increasingly a software problem rather than raw hardware. By rethinking how model state is mapped onto existing consumer resources, the project opens a path for affordable, high‑performance AI inference on personal machines, potentially democratizing access to the most capable language models.