PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantization
PagedWeight shifts MoE expert weights between precisions at runtime to make room for KV cache without giving up accuracy.
The paper targets memory-sensitive MoE serving, where model weights and an expanding KV cache compete for GPU memory. Its method dynamically quantizes expert weights and chooses precision based on the quality, memory, and latency tradeoff. The authors report FP16-equivalent accuracy with up to 72.0% GPU memory savings and 1.94x throughput improvement. Against other quantization baselines, it improves quality by up to 39.3% at a similar memory budget, with at most 4.1% throughput loss. ArXiv · AI/CL/LG's note
The paper targets memory-sensitive MoE serving, where model weights and an expanding KV cache compete for GPU memory. Its method dynamically quantizes expert weights and chooses precision based on the quality, memory, and latency tradeoff. The authors report FP16-equivalent accuracy with up to 72.0% GPU memory savings and 1.94x throughput improvement. Against other quantization baselines, it improves quality by up to 39.3% at a similar memory budget, with at most 4.1% throughput loss. ArXiv · AI/CL/LG's note
score 5