CoPre-MoE: Consolidating and Preserving MoE for Multimodal Continual Instruction Tuning
The authors report better continual-learning performance with less parameter overhead by merging overlapping experts and compressing task-specific ones.
CoPre-MoE targets multimodal models that must learn new tasks without forgetting earlier capabilities. It consolidates overlapping experts into a fixed-capacity generic pool and compresses remaining specialists using shared projection bases and compact task-specific cores. Its router freezes historical retrieval keys and penalizes overlap among new keys to help preserve earlier learning. At inference, it combines both expert pools without task labels; the abstract provides no numerical results.
ArXiv · AI/CL/LG's note
CoPre-MoE targets multimodal models that must learn new tasks without forgetting earlier capabilities. It consolidates overlapping experts into a fixed-capacity generic pool and compresses remaining specialists using shared projection bases and compact task-specific cores. Its router freezes historical retrieval keys and penalizes overlap among new keys to help preserve earlier learning. At inference, it combines both expert pools without task labels; the abstract provides no numerical results.
ArXiv · AI/CL/LG's note
score 4