Megadose The AI curve, compressed.

CoPre-MoE: Consolidating and Preserving MoE for Multimodal Continual Instruction Tuning

· ArXiv · AI/CL/LG ·
The authors report better continual-learning performance with less parameter overhead by merging overlapping experts and compressing task-specific ones.

CoPre-MoE targets multimodal models that must learn new tasks without forgetting earlier capabilities. It consolidates overlapping experts into a fixed-capacity generic pool and compresses remaining specialists using shared projection bases and compact task-specific cores. Its router freezes historical retrieval keys and penalizes overlap among new keys to help preserve earlier learning. At inference, it combines both expert pools without task labels; the abstract provides no numerical results.

ArXiv · AI/CL/LG's note

score 4

Categories: Research