Multi-View MoE: Letting Every Router Shape Expert Computation
Multi-View MoE raised the nine-task average by 1.18 percentage points while preserving expert FLOPs and communication volume.
The researchers tested a 24-layer 10B-A1B model trained on 708 billion tokens. Multiple routers combine their scores to select experts, then reuse those preferences to mix learned representations into each selected expert’s input. Each selected expert runs once, keeping the active-expert count fixed. Multiple routers alone delivered a 0.6-percentage-point gain and improved alignment between routing scores and evaluated expert utility.
ArXiv · AI/CL/LG's note
The researchers tested a 24-layer 10B-A1B model trained on 708 billion tokens. Multiple routers combine their scores to select experts, then reuse those preferences to mix learned representations into each selected expert’s input. Each selected expert runs once, keeping the active-expert count fixed. Multiple routers alone delivered a 0.6-percentage-point gain and improved alignment between routing scores and evaluated expert utility.
ArXiv · AI/CL/LG's note
score 4