Reduced Matrix Multiplication: Input-Adaptive Matrix-Product Reduction for LLM Inference
RMM cuts Transformer inference work by dropping less informative matrix-product slices at runtime, without retraining or changing weights.
The paper reports a controllable accuracy-efficiency trade-off using a retention ratio. Tests span 1B to 70B language models, with robustness under moderate reduction across discriminative, generation, and long-context settings. The authors find attention computations are much easier to reduce than MLP components. A100 benchmarks with custom kernels show the savings can become real runtime gains, especially on longer sequences. ArXiv · AI/CL/LG's note
The paper reports a controllable accuracy-efficiency trade-off using a retention ratio. Tests span 1B to 70B language models, with robustness under moderate reduction across discriminative, generation, and long-context settings. The authors find attention computations are much easier to reduce than MLP components. A100 benchmarks with custom kernels show the savings can become real runtime gains, especially on longer sequences. ArXiv · AI/CL/LG's note
score 5