NeuPAT: Neuron-aware Plasticity Allocation Tuning for Language-Preserving MLLMs
NeuPAT tries to stop multimodal tuning from erasing a pretrained model’s language skills.
The paper says neurons adapt unevenly during multimodal learning: some are language-sensitive, while others can absorb multimodal knowledge more readily. NeuPAT adds a small probing stage, then applies neuron-wise update constraints during multimodal instruction tuning. Across tested LLM families, the authors report recovering 94.5% of the language degradation from vanilla tuning across 11 language benchmarks while keeping comparable multimodal performance. HF Daily Papers' note
The paper says neurons adapt unevenly during multimodal learning: some are language-sensitive, while others can absorb multimodal knowledge more readily. NeuPAT adds a small probing stage, then applies neuron-wise update constraints during multimodal instruction tuning. Across tested LLM families, the authors report recovering 94.5% of the language degradation from vanilla tuning across 11 language benchmarks while keeping comparable multimodal performance. HF Daily Papers' note
score 5