Janas: Run LLMs larger than RAM by streaming experts from SSD
Janas reports Qwen3-Next-80B running at 37 tok/s with drafts on a 32 GB laptop by keeping expert weights on SSD.
The project says Janas-LLM streams the needed mixture-of-experts weights from NVMe storage and caches recurring experts in RAM. Its benchmark table shows Janas beating llama.cpp on the same laptop across the compared models, including 24 tok/s versus 2.8 tok/s for Qwen3-Next-80B without drafts. The author stresses this is an early experiment, tested mainly on one Intel Core Ultra 9 laptop, and warns it can consume free memory and hit the disk hard. It ships Linux and Windows x86-64 builds, with `janas-bench` meant to measure performance on other machines. HN · LLMs' note
The project says Janas-LLM streams the needed mixture-of-experts weights from NVMe storage and caches recurring experts in RAM. Its benchmark table shows Janas beating llama.cpp on the same laptop across the compared models, including 24 tok/s versus 2.8 tok/s for Qwen3-Next-80B without drafts. The author stresses this is an early experiment, tested mainly on one Intel Core Ultra 9 laptop, and warns it can consume free memory and hit the disk hard. It ships Linux and Windows x86-64 builds, with `janas-bench` meant to measure performance on other machines. HN · LLMs' note
score 5