Fuse-All MoE Inference Kernel with Load Balancing on TPUs
The integrated system delivers up to 34.5% higher serving throughput than Megablox EP16, with serving-stack optimizations included.
Fuse-All MoE combines token collection, expert projections, gated activation and output writes in one TPU kernel, keeping intermediate computation on chip. Scheduling and expert redistribution bring a measured skewed prefill workload from 2.3× to 1.09× the fixed balanced-workload reference time. For decode workloads of 32–512 tokens, sharing active expert computation across each chip’s two TensorCore devices cuts kernel latency by up to 21.3% versus fused EP16.
ArXiv · AI/CL/LG's note
Fuse-All MoE combines token collection, expert projections, gated activation and output writes in one TPU kernel, keeping intermediate computation on chip. Scheduling and expert redistribution bring a measured skewed prefill workload from 2.3× to 1.09× the fixed balanced-workload reference time. For decode workloads of 32–512 tokens, sharing active expert computation across each chip’s two TensorCore devices cuts kernel latency by up to 21.3% versus fused EP16.
ArXiv · AI/CL/LG's note
score 5