The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction
Edge0 claims a 35B mixture-of-experts model can run from SSD on a single 24GB machine at 20 tokens per second.
The paper says the bottleneck is not sparse compute, but keeping the right expert weights available in time. Its fix is a per-layer prerouter that predicts the next layer’s routing one token ahead, so SSD reads can be staged before they are needed. A recovery LoRA is used to regain quality lost from int4 quantization and replacing the usual router path. The authors report peak active memory under 3GiB, benchmark results within a few points of an fp16 teacher, and open-source framework, checkpoints, and adapters. HF Daily Papers' note
The paper says the bottleneck is not sparse compute, but keeping the right expert weights available in time. Its fix is a per-layer prerouter that predicts the next layer’s routing one token ahead, so SSD reads can be staged before they are needed. A recovery LoRA is used to regain quality lost from int4 quantization and replacing the usual router path. The authors report peak active memory under 3GiB, benchmark results within a few points of an fp16 teacher, and open-source framework, checkpoints, and adapters. HF Daily Papers' note
score 5