IntBMoE: Integrating Block-Level Conditioning into Expert Composition for Full-Participation Mixture-of-Experts
The paper claims a MoE design that keeps full expert participation while limiting compute and memory costs.
IntBMoE composes experts from learned block codebooks, then routes each token through only a few blocks. The authors say this separates participation, execution, and materialization, which current sparse, dense, and parameter-merging MoE designs do not. Experiments cover image classification, language modeling, and sequential recommendation. They also report deployment in AMap’s generative recommendation system, with a 2.4% relative UVCTR gain under a 60ms latency budget. HF Daily Papers' note
IntBMoE composes experts from learned block codebooks, then routes each token through only a few blocks. The authors say this separates participation, execution, and materialization, which current sparse, dense, and parameter-merging MoE designs do not. Experiments cover image classification, language modeling, and sequential recommendation. They also report deployment in AMap’s generative recommendation system, with a 2.4% relative UVCTR gain under a 60ms latency budget. HF Daily Papers' note
score 5