Megadose The AI curve, compressed.

Fuse-All MoE Inference Kernel with Load Balancing on TPUs

· ArXiv · AI/CL/LG ·
The integrated system delivers up to 34.5% higher serving throughput than Megablox EP16, with serving-stack optimizations included.

Fuse-All MoE combines token collection, expert projections, gated activation and output writes in one TPU kernel, keeping intermediate computation on chip. Scheduling and expert redistribution bring a measured skewed prefill workload from 2.3× to 1.09× the fixed balanced-workload reference time. For decode workloads of 32–512 tokens, sharing active expert computation across each chip’s two TensorCore devices cuts kernel latency by up to 21.3% versus fused EP16.

ArXiv · AI/CL/LG's note

score 5

Categories: Research