about
FreeToken: Datacenter-class intelligence on the hardware you own (arxiv.org)
3 points by lioeters 15 days ago | hide | past | pdf | 3 comments on HN

In plain words: FreeToken keeps a huge model's many specialist parts moving between memory, disk, and the GPU as space frees up, instead of sticking to one fixed offloading plan. That lets a single workstation GPU run a 753B-parameter model that normally needs a datacenter.

Abstract · FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution

Frontier open-weight models are increasingly available, but serving them still largely assumes datacenter infrastructure. We present FreeToken, an edge-native MoE serving system that treats a personal machine not as a small GPU, but as a unified, elastic inference platform. FreeToken co-designs the full serving stack, including model layout and loading, expert residency, CPU--GPU execution, agentic state reuse, and runtime memory management, around two realities of local AI: agent workloads continuously change their execution pattern, and edge hardware exposes heterogeneous resources whose balance differs from machine to machine. Rather than committing to a fixed offloading strategy, FreeToken continuously maps computation and model state onto the resources actually available. FreeToken supports more than 20 MoE models and real coding and tool-using agents across hardware ranging from an 8GB laptop GPU to a single workstation GPU. More importantly, it changes what these machines can practically serve, from a 35B model on a laptop to a 284B model on a gaming desktop and the 753B GLM-5.2 on a single workstation GPU. FreeToken turns open weights into deployable local software, making the machines users already own a practical platform for frontier-scale intelligence. We release the system at flashml.ai.

Shuo Yang, Xiaoze Fan, Melissa Pan, Haocheng Xi, Zhe Wang, Shanlin Sun, Kurt Keutzer, Song Han, Matei Zaharia, Chenfeng Xu, Ion Stoica
arXiv:2608.16157 · cs.DC · submitted Aug 17, 2026
abstract · pdf · html

add comment on HN
Also discussed: Aug 2026 (2 points, 0 comments) · Aug 2026 (1 point, 0 comments) · Aug 2026 (3 points, 0 comments) · Aug 2026 (4 points, 0 comments)

Source: https://github.com/FlashML-org/FreeToken

> FreeToken is an edge-native Mixture-of-Experts (MoE) serving engine designed for running frontier-scale open-weight models on personal and consumer hardware. It treats heterogeneous edge resources—GPUs, CPUs, host memory, and interconnects—as a unified, elastic inference platform.

Nothing new, the models have to fit in RAM+VRAM contrary to the expectation you might have from the title.
Sounds almost too good to be true. Has anyone tried it?