Spend Experts Where You Are Unsure: Confidence-Adaptive Routing for Mixture-of-Experts LoRA
CARE varies the number of LoRA experts per token using the router’s own confidence signal.
The paper argues fixed top-k MoE-LoRA wastes compute on easy tokens and underserves ambiguous ones. CARE activates experts until their router weights reach a threshold, with an added check when experts disagree. A budget thermostat keeps average expert use at a chosen target, without extra parameters or extra forward passes. In reported tests across commonsense, math, code, and knowledge tasks, it beats fixed top-k at matched compute and matches fixed k=4 while using fewer experts. ArXiv · AI/CL/LG's note
The paper argues fixed top-k MoE-LoRA wastes compute on easy tokens and underserves ambiguous ones. CARE activates experts until their router weights reach a threshold, with an added check when experts disagree. A budget thermostat keeps average expert use at a chosen target, without extra parameters or extra forward passes. In reported tests across commonsense, math, code, and knowledge tasks, it beats fixed top-k at matched compute and matches fixed k=4 while using fewer experts. ArXiv · AI/CL/LG's note
score 4