Modality-Aware Expert Pruning for MoE-Based Multimodal Large Language Models
MAEP cuts roughly 27 GB of expert weights, bringing an approximately 58 GB model’s weights within a single 40 GB GPU’s capacity.
The training-free method assesses experts separately for vision and language in a single forward pass. It uses text importance to preserve experts and low-attention vision tokens to identify pruning candidates, ranking experts across layers. Its scoring metric measures how removing an expert changes the MoE output, accounting for redistribution. Across three multimodal models, MAEP beats modality-agnostic baselines on most multimodal benchmarks while remaining competitive on text-only tasks.
HF Daily Papers' note
The training-free method assesses experts separately for vision and language in a single forward pass. It uses text importance to preserve experts and low-attention vision tokens to identify pruning candidates, ranking experts across layers. Its scoring metric measures how removing an expert changes the MoE output, accounting for redistribution. Across three multimodal models, MAEP beats modality-agnostic baselines on most multimodal benchmarks while remaining competitive on text-only tasks.
HF Daily Papers' note
score 5