JustFit: 200K-Token LLM Serving on a 24 GiB Laptop with Just-in-Time State Management
JustFit reports 212,992-token single-request serving on a 24 GiB M4 Pro MacBook.
The paper describes an MLX inference runtime using compressed KV execution, component swapping, and state-preserving serving transitions. In tests with Qwen3.8-27B MXFP4, it completed 196,608 input tokens and 16,384 output tokens across three runs. That was 6.93x the completed single-request context of the mlx-vlm baseline cited in the abstract. A separate two-request run held 229,376 positions in aggregate. ArXiv · AI/CL/LG's note
The paper describes an MLX inference runtime using compressed KV execution, component swapping, and state-preserving serving transitions. In tests with Qwen3.8-27B MXFP4, it completed 196,608 input tokens and 16,384 output tokens across three runs. That was 6.93x the completed single-request context of the mlx-vlm baseline cited in the abstract. A separate two-request run held 229,376 positions in aggregate. ArXiv · AI/CL/LG's note
score 6