MoExtend: Tuning New Experts for Modality and Task Extension

Shanshan Zhong, Shanghua Gao, Zhongzhan Huang, Wushao Wen, Marinka Zitnik, Pan Zhou

Abstract

Large language models (LLMs) excel in various tasks but are primarily trained on text data, limiting their application scope. Expanding LLM capabilities to include vision-language understanding is vital, yet training them on multimodal data from scratch is challenging and costly. Existing instruction tuning methods, e.g., LLAVA, often connects a pretrained CLIP vision encoder and LLMs via fully fine-tuning LLMs to bridge the modality gap. However, full fine-tuning is plagued by catastrophic forgetting, i.e., forgetting previous knowledge, and high training costs particularly in the era of increasing tasks and modalities. To solve this issue, we introduce MoExtend, an effective framework designed to streamline the modality adaptation and extension of Mixture-of-Experts (MoE) models. MoExtend seamlessly integrates new experts into pre-trained MoE models, endowing them with novel knowledge without the need to tune pretrained models such as MoE and vision encoders. This approach enables rapid adaptation and extension to new modal data or tasks, effectively addressing the challenge of accommodating new modalities within LLMs. Furthermore, MoExtend avoids tuning pretrained models, thus mitigating the risk of catastrophic forgetting. Experimental results demonstrate the efficacy and efficiency of MoExtend in enhancing the multimodal capabilities of LLMs, contributing to advancements in multimodal AI research.

Introduction

As AI moves toward multimodal capabilities, the standard approach involves fine-tuning massive text-trained Large Language Models (LLMs) on image-text pairs. However, this paradigm faces two major hurdles: catastrophic forgetting and prohibitive computational costs. When a model is forced to learn a new skill, such as visual description, it often "overwrites" its existing logic or coding knowledge. Furthermore, fully fine-tuning models with billions of parameters is slow and resource-intensive. While Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA exist, they often struggle to bridge the fundamental "modality gap" between text and pixels. MoExtend addresses these challenges by leveraging the modular nature of Mixture-of-Experts (MoE) architectures.

Methods

MoExtend introduces a three-stage pipeline to adapt MoE models to new tasks efficiently, as illustrated in the framework overview.

The three-stage pipeline of MoExtend: Alignment, Extension, and Fine-tuning.
  1. Alignment Stage: A vision encoder (CLIP) is connected to the LLM via a trainable MLP projector to align visual features with the model's text space.
  2. Extension Stage (The "Extender"): Rather than adding experts to every layer, MoExtend uses a distribution shift metric. By performing minimal tuning on gating layers, the framework identifies which layers' expert-selection probabilities change most significantly. New experts are then added only to the top 50% of these sensitive layers.
  3. Fine-tuning Stage: New experts are initialized using weights from existing experts that responded most strongly to the new data (a "warm start"). Additionally, lightweight Calibration Modules are introduced to ensure the original feature distribution remains stable, preventing new experts from degrading existing knowledge.
Detailed architecture of the MoE layer extension, including the new expert and calibration modules.

Results and Discussion

MoExtend was evaluated using the LLaVA 1.5 datasets across nine multimodal benchmarks (e.g., ScienceQA, TextVQA, MME) and several text-only benchmarks (e.g., ARC-e, MMLU) to assess forgetting.

Efficiency: MoExtend achieves a roughly 6x training speedup compared to full fine-tuning, reducing instruction tuning time from 200 hours to 30 hours.

Performance: Despite having only 3B activated parameters, the MoExtend model outperformed the 13B parameter LLaVA-1.5 model on ScienceQA (73.8 vs 71.6) and the MME benchmark (1710.1 vs 1531.3).

Knowledge Preservation: On pure text benchmarks, MoExtend showed a negligible performance drop of only 0.41%, whereas full fine-tuning caused a 3.30% drop. This confirms that freezing the original model and using selective expansion effectively mitigates catastrophic forgetting.

Conclusion

MoExtend provides a scalable and efficient framework for extending MoE models to new modalities. By selectively adding and calibrating new experts, it achieves superior multimodal performance while preserving original text-based reasoning capabilities. Future work will explore scaling to larger models and sequential multi-modality extension in continual learning settings.

Summary updated

Abstract

Large language models (LLMs) excel in various tasks but are primarily trained on text data, limiting their application scope. Expanding LLM capabilities to include vision-language understanding is vital, yet training them on multimodal data from scratch is challenging and costly. Existing instruction tuning methods, e.g., LLAVA, often connects a pretrained CLIP vision encoder and LLMs via fully fine-tuning LLMs to bridge the modality gap. However, full fine-tuning is plagued by catastrophic forgetting, i.e., forgetting previous knowledge, and high training costs particularly in the era of increasing tasks and modalities. To solve this issue, we introduce MoExtend, an effective framework designed to streamline the modality adaptation and extension of Mixture-of-Experts (MoE) models. MoExtend seamlessly integrates new experts into pre-trained MoE models, endowing them with novel knowledge without the need to tune pretrained models such as MoE and vision encoders. This approach enables rapid adaptation and extension to new modal data or tasks, effectively addressing the challenge of accommodating new modalities within LLMs. Furthermore, MoExtend avoids tuning pretrained models, thus mitigating the risk of catastrophic forgetting. Experimental results demonstrate the efficacy and efficiency of MoExtend in enhancing the multimodal capabilities of LLMs, contributing to advancements in multimodal AI research. Code: https://github.com/zhongshsh/MoExtend.
BibTeX
@inproceedings{zhong2024moextend,
  author = {Shanshan Zhong and Shanghua Gao and Zhongzhan Huang and Wushao Wen and Marinka Zitnik and Pan Zhou},
  title = {{MoExtend: Tuning New Experts for Modality and Task Extension}},
  booktitle = {Student Research Workshop @ ACL},
  year = {2024},
  url = {https://openreview.net/forum?id=VXF1uX8ClX},
}
Figure