StrataFlux: Memory-Hierarchy-Native Low-Bit LLM Inference
StrataFlux raises 2-bit Qwen3-8B’s AIME accuracy from 11.0% to 47.1% without increasing GPU memory use, the authors report.
The framework streams additional precision from host RAM and SSDs to recover accuracy lost when compressing model weights and the KV cache. It overlaps transfers with computation and reuses fetched precision across decoding steps. On RTX 5090, it lowers 2-bit Llama-3.1-8B-Instruct’s WikiText-2 perplexity from 9.46 to 8.40 while retaining 95.4% of the quantized baseline’s throughput. At 128K context, another configuration matches the reported perplexity and speed of vLLM’s 4-bit-weight, 8-bit-KV setup using approximately half the GPU memory.
ArXiv · AI/CL/LG's note
The framework streams additional precision from host RAM and SSDs to recover accuracy lost when compressing model weights and the KV cache. It overlaps transfers with computation and reuses fetched precision across decoding steps. On RTX 5090, it lowers 2-bit Llama-3.1-8B-Instruct’s WikiText-2 perplexity from 9.46 to 8.40 while retaining 95.4% of the quantized baseline’s throughput. At 128K context, another configuration matches the reported perplexity and speed of vLLM’s 4-bit-weight, 8-bit-KV setup using approximately half the GPU memory.
ArXiv · AI/CL/LG's note
score 5