Multimodal Synergy-Aware Expert Pruning for MoE-Based Multimodal LLMs
Lists
Only you see this
Abstract
MoE-based multimodal LLMs integrate text, image, and audio modalities to achieve powerful multimodal understanding, but redundant experts across modalities result in high memory consumption and increased computational complexity. Existing expert pruning methods are limited to text-only MoE LLMs and cannot capture the cross-modal synergy of experts, leading to performance loss in multimodal tasks. To address this gap, we propose Multimodal Synergy-Aware Expert Pruning (MSAEP), a specialized framework for MoE-based MLLMs that prunes redundant experts while preserving cross-modal alignment and synergy. MSAEP includes two key innovations: 1) a Multimodal Synergy Expert Importance Score (MSEIS) that quantifies each expert’s contribution to single-modal and cross-modal tasks by combining modality-specific activation frequency and cross-modal synergy coefficients; 2) a Synergy-Aware Pruning Strategy (SAPS) that prunes experts with low MSEIS scores, ensuring the remaining experts cover all modalities and maintain effective cross-modal collaboration. MSAEP also features a Cross-Modal Router Tuning (CMRT) module to rebalance expert activation across modalities after pruning. Experiments on multimodal benchmarks (MMBench, Flickr30K, VQA-v2) using the MoE-LLaVA model show that MSAEP prunes 45% of experts while maintaining 98.9% of the full-model’s performance on multimodal tasks, reducing memory overhead by 42% and inference latency by 37%.
BibTeX
@misc{li2026multimodal,
author = {Lujun Li},
title = {{Multimodal Synergy-Aware Expert Pruning for MoE-Based Multimodal LLMs}},
year = {2026},
url = {https://openreview.net/forum?id=eYUu6OcthR},
howpublished = {OpenReview},
}Loading related content...