Frontier open-weight fashions are delivery quicker than the {hardware} assumptions round them. Kimi-K3, GLM-5.2 and DeepSeek-V4-Flash are closing the aptitude hole with proprietary methods, however releasing parameters solely determines who can receive a mannequin — not who can afford to run it. Serving them nonetheless assumes datacenter-class GPU clusters, and as agentic workloads push inference demand up, that price lands hardest on particular person builders and small groups. In the meantime, greater than 100 million client machines already carry discrete GPUs. A group of researchers from UC Berkeley and UT Austin suggest FreeToken. The analysis group argued the lacking piece isn’t {hardware} however a serving system: it treats a private machine as a unified, elastic inference platform fairly than a small GPU, and constantly maps computation and mannequin state onto no matter GPU, CPU, reminiscence and interconnect bandwidth the machine really has. The result’s a 35B mannequin at interactive velocity on an 8 GB laptop computer GPU, 284B on a gaming desktop, and the 753B GLM-5.2 on a single workstation card.
Is it deployable?
Sure, FreeToken is Apache-2.0 on GitHub, printed on PyPI as freetoken v0.1.2 (uv pip set up "freetoken[accel]"), and shipped as a one-click desktop app for Home windows and Linux at flashml.ai. The CLI targets Linux x86_64 with an NVIDIA GPU on driver r580+ (CUDA 13). ft serve exposes OpenAI- and Anthropic-compatible endpoints on port 1919, and ft launch claude wires up Claude Code, Codex, OpenCode or OpenClaw in opposition to your personal field.
Who it suits: solo builders, startups and SMB engineering groups whose agent token payments already exceed the price of a GPU they personal; enterprises ought to deal with it as an air-gapped or regulated-workload path, not a datacenter substitute. Strongest trade match: healthcare and authorized (knowledge by no means leaves the machine), protection, finance, and IP-heavy R&D. Typical functions: native coding brokers, non-public code overview, offline contract evaluation, synthetic-data era, batch evals.
The hole it targets
Combination-of-Specialists makes native frontier inference arithmetically possible. DeepSeek-V4-Flash prompts 6 of 256 routed specialists in every of 43 layers, so solely 13B of its 284B parameters take part in any single token. Sparsity doesn’t shrink the skilled pool, although — at FP4 the complete set is roughly 140 GB, so inactive specialists sit in host reminiscence and enter the execution path on demand.
The analysis group isolates three failure modes in current engines (llama.cpp, KTransformers, Ollama, MoE-Infinity):
- Prefill destroys sparsity: Hundreds of tokens per layer route to almost the entire skilled set, so a prefill move streams your complete pool throughout PCIe — about two seconds on an RTX 5090, 5 on PCIe 4.0 desktops, ten or extra on the x8 hyperlinks widespread in laptops.
- Static placement misses decode site visitors: llama.cpp assigns MoE tensors at load time; KTransformers pins a “scorching” subset. Routing shifts each token, so most skilled evaluations fall to the CPU whereas the GPU and the PCIe hyperlink sit idle.
- Client CPUs can not carry the rest: Twin-channel DDR5 delivers 80–90 GB/s in opposition to the 1–1.8 TB/s an RTX 4090 or 5090 attracts from on-package reminiscence.
Three mechanisms
- Bandwidth-adaptive execution (the q* coverage): As a result of DMA transfers and CPU skilled execution learn from the identical host-memory subsystem, a saturated PCIe hyperlink leaves a residual bandwidth of
B_H − B_P. FreeToken splits every step’s m cache misses accordingly:q* ≈ m × B_P / B_Hspecialists are crammed into the GPU cache, the remaining are computed in place on the CPU, and the 2 partial sums merge precisely — no approximation, no router modification. Each bandwidths are profiled on the deployed machine (ft bench bw), which issues: measured B_P:B_H is 52.7:77.3 on an RTX 5090 server however 11.8:47.5 on a 4060 laptop computer. - Semantic-aware caching: Throughout prefill, full-layer double buffering streams layer l+1 whereas the GPU computes layer l. Recurrent-state checkpoints are anchored at special-token boundaries — pondering blocks, instrument calls, instrument outputs — exactly the place agent harnesses truncate context, so an edit re-prefills solely the brand new suffix. Throughout decode, a shared LRU skilled cache spanning all MoE layers follows the router as an alternative of a placement frozen at load time.
- Elastic reminiscence administration: At scheduler protected factors the GPU skilled cache is rebuilt beneath a revised VRAM finances with out restarting the engine or reloading the host pool. Specialists are learn from disk straight into their remaining host format, then pinned; no GPU warmup is required as a result of the primary request is served with a chilly cache.
Outcomes
On an RTX 5090, FreeToken sustains 77–83 tok/s on Qwen3.6-35B-A3B (BF16) and 22–25 tok/s on DeepSeek-V4-Flash (MXFP4) — 1.5–2.3× the strongest baseline, with decode staying inside 12% of the single-turn charge throughout three agentic workloads. Worst-case TTFT stays beneath 44 s in each cell; llama.cpp hits 232 s, Ollama 179 s and KTransformers 946 s someplace within the matrix, previous the purpose the place agent purchasers day trip.
At equal cache capability (37% of the Qwen3.6 pool), the worldwide LRU misses 16% of decode-time skilled reads in opposition to 41% for KTransformers and 62% for llama.cpp. On an 8 GB RTX 4060 laptop computer the NVFP4 construct serves 35B at 39.3 tok/s — above the 33 tok/s median decode velocity measured for Codex in manufacturing traces. On a single RTX PRO 6000, GLM-5.2 (753B, 40B energetic) runs at 14.9 tok/s versus llama.cpp’s 7.3.
Information Examine
Actuality Examine · FlashML FreeToken
INFLATION SCORE 59/100
3Verified
9Self-rep.
4Deceptive
0Contradicted
0Not discovered
Rating system: 8 × deceptive + 15 × contradicted + 3 × self-reported, capped at 100. The rating is pushed by the self-reported column, not by dishonesty — the code went public six days earlier than this audit, so no impartial copy exists but.
Declare desk · 16 claims
| Declare | Their quantity | Unbiased verify | Verdict & supply |
|---|---|---|---|
| Decode, Qwen3.6-35B-A3B BF16, RTX 5090 | 77–83 tok/s | None discovered | SELF-REPORTEDPaper Fig 3 |
| Decode, DeepSeek-V4-Flash MXFP4, RTX 5090 | 22–25 tok/s | None discovered | SELF-REPORTEDPaper Fig 3 |
| Decode speedup vs strongest baseline | 1.5–2.3× | Recomputes precisely from Fig 3 | SELF-REPORTEDPaper §5.2 |
| Decode stability throughout agent workloads | inside 12% of W1 | None discovered | SELF-REPORTEDPaper §5.2 |
| Worst-case TTFT vs baselines | <44 s vs 232 / 179 / 946 s | None discovered | SELF-REPORTEDPaper §5.2 |
| 4060 laptop computer “exceeds Codex median 33 tok/s” | 39.3 vs 33 | TraceLab 33.9 is normalized; Codex pure decode median 57.1, w.avg 61.0 | MISLEADINGarXiv:2606.30560 |
| Laptop computer is “92% of the RTX 4090 charge” | 39.3 / 42.9 | Arithmetic right, however 39.3 is NVFP4 and 42.9 is BF16 | MISLEADINGPaper Fig 5 |
| GLM-5.2 753B on one RTX PRO 6000 | 14.9 vs llama.cpp 7.3 | None discovered | SELF-REPORTEDPaper §5.3 |
| Cross-hardware lead, 5 client methods | 1.3–2.1× | Recomputes precisely from Fig 5 | SELF-REPORTEDPaper Fig 5 |
| Decode skilled miss charge at equal capability | 16% / 39% | None discovered; hint replay, not stay serving | SELF-REPORTEDPaper Fig 4b |
| Prefill 8,192-tok chunk; overlap penalty | 1.19–1.22 s; 19/25/26% | None discovered | SELF-REPORTEDPaper Fig 4a |
| “753B on a single workstation GPU” framing | 1 GPU | True for VRAM; hosts carry 512 GiB and 192 GB DRAM | MISLEADINGPaper Desk 1 |
| Baselines run at 6 CPU threads on rented servers | 6 threads | KTransformers’ core contribution is many-core AMX CPU kernels | MISLEADINGPaper §5.1 |
| “Helps greater than 20 MoE fashions” | 20+ | Public docs/fashions.md itemizes ~17 known-good MoE checkpoints | SELF-REPORTEDrepo docs |
| License and distribution | Apache-2.0, PyPI v0.1.2 | LICENSE file and PyPI JSON API each verify | VERIFIEDGitHub, PyPI |
| Client discrete-GPU set up base (Steam foundation) | ~72% NVIDIA; 4060 Laptop computer 3.81% | Matches Valve June 2026 survey per a number of retailers | VERIFIEDValve, Jul 2026 |
| Inside arithmetic throughout summary and §5 | all ratios | Each printed ratio recomputes from Figures 3 and 5; zero errors | VERIFIEDrecomputed |
Flags defined
Denominator video games — the Codex comparability mixes two metricsFreeToken experiences decode throughput and TTFT individually, so its 39.3 tok/s is a pure decode charge. TraceLab’s 33.9 tok/s is a normalized charge that folds per-step TTFT into decode; the identical paper places Codex’s pure decode median at 57.1 tok/s. Like-for-like, 39.3 doesn’t exceed Codex — it’s roughly two thirds of it.
Denominator video games — “92% of the RTX 4090 charge” compares 4-bit to 16-bitThe 39.3 tok/s laptop computer determine is the NVFP4 construct; the 42.9 tok/s RTX 4090 determine is BF16. Disclosed within the Determine 5 caption, however the prose states the ratio with out the precision caveat.
Denominator video games — “single GPU” omits the host requirementThe 753B GLM-5.2 tier sits behind 512 GiB of DDR5 on a Xeon Platinum 8559C; the 284B “gaming desktop” carries 192 GB. One GPU is correct. One machine at client costs isn’t what these configurations describe.
Settings mismatch — baselines capped at 6 CPU threadsDisclosed and defensible: the paper caps rented dual-socket servers to emulate edge hosts and validates on two actual edge machines at full threads. However KTransformers is constructed round AMX-optimized many-core CPU skilled execution, so learn its column as “KTransformers on an edge-class host,” not as its ceiling.
Counterweight — the arithmetic is clearEach ratio within the summary and outcomes recomputes accurately: 1.81 / 1.87 / 2.10 / 2.25× for Qwen3.6, 1.92 / 1.84 / 1.52 / 1.65× for DeepSeek-V4-Flash, 2.04× for GLM-5.2. No inflated rounding, no unexplained gaps between figures and prose.
Key takeaways
- The paper is arithmetically clear — each printed ratio recomputes from its personal figures.
- Nothing is independently reproduced but; 9 of 16 claims are self-reported by necessity, not evasion.
- Sharpest flag: 39.3 tok/s beats Codex’s normalized 33.9, not its pure decode median of 57.1.
- “Single GPU” headlines quietly require 192–512 GB of host DRAM.
- Baseline KTransformers runs at 6 CPU threads, beneath the many-core AMX config it targets.
Actuality Examine by Marktechpost · verified Aug 23, 2026
Key Takeaways
- FreeToken splits MoE cache misses between PCIe fills and CPU execution utilizing measured bandwidths, not a hard and fast offload rule.
- Skilled output stays bit-exact — no router modifications, no skilled substitution, no precision rest.
- 1.5–2.3× decode throughput over llama.cpp, Ollama and KTransformers, with tail TTFT beneath 44 s.
- 35B at 39.3 tok/s on an 8 GB laptop computer GPU; 753B GLM-5.2 on one workstation GPU.
- Apache-2.0, on PyPI and as a Home windows/Linux desktop app — deployable this afternoon.
Try the PAPER, GITHUB REPO and PROJECT. Additionally, be at liberty to comply with us on Twitter and don’t overlook to hitch our 150k+ML SubReddit and Subscribe to our E-newsletter. Wait! are you on telegram? now you’ll be able to be part of us on telegram as properly.
Have to associate with us for selling your GitHub Repo OR Hugging Face Web page OR Product Launch OR Webinar and so on.? Join with us
Asif Razzaq is the CEO of Marktechpost Media Inc.. As a visionary entrepreneur and engineer, Asif is dedicated to harnessing the potential of Synthetic Intelligence for social good. His most up-to-date endeavor is the launch of an Synthetic Intelligence Media Platform, Marktechpost, which stands out for its in-depth protection of machine studying and deep studying information that’s each technically sound and simply comprehensible by a large viewers. The platform boasts of over 2 million month-to-month views, illustrating its reputation amongst audiences.
