On Domain-Adaptive Post-Training for Multimodal Large Language Models

Daixuan Cheng, Shaohan Huang, Ziyu Zhu, Xintong Zhang, Xin Zhao, Zhongzhi Luan, Bo Dai, Zhenliang Zhang

Introduction

General-purpose Multimodal Large Language Models (MLLMs), such as LLaVA and Qwen-VL, have demonstrated remarkable versatility in common visual-linguistic tasks. However, their performance often degrades significantly when applied to specialized domains like biomedicine, food science, or remote sensing. Adapting these models to niche fields is traditionally hindered by data scarcity, the high cost of expert annotation, and privacy concerns associated with closed-source synthesis APIs. Furthermore, standard multi-stage training pipelines often suffer from "catastrophic forgetting," where the model loses foundational domain knowledge during the transition from captioning to instruction-following tasks.

Methodology: The AdaMLLM Framework

To address these challenges, we introduce AdaMLLM, a systematic approach for domain-adaptive post-training. The framework consists of two primary components: a "generate-then-filter" data synthesis pipeline and a streamlined single-stage training process.

Visual Instruction Synthesizer

Rather than relying on manual rules or proprietary models, AdaMLLM fine-tunes an open-source MLLM to serve as a Visual Instruction Synthesizer. This synthesizer transforms domain-specific image-caption pairs into high-quality "task triplets" comprising a complex instruction, an informative reasoning-based response, and a precise final answer. To ensure data quality, a consistency-based filter is employed; an LLM verifies the logical alignment between the informative response and the precise answer, discarding contradictory samples.

The AdaMLLM framework: (A) Fine-tuning the Visual Instruction Synthesizer and (B) the synthesis and filtering pipeline for post-training.

Single-Stage Post-Training

Unlike the conventional two-stage approach (pre-training on captions followed by instruction tuning), AdaMLLM utilizes a single-stage post-training strategy. By mixing image-captioning tasks with synthetic instruction-following tasks throughout the training process, the model maintains task diversity and prevents over-specialization. This approach ensures that the model retains the ability to generate detailed descriptions while mastering complex domain-specific reasoning.

Experimental Setup and Results

The AdaMLLM framework was evaluated across three high-impact domains: Biomedicine (using PubMed Central data), Food (using Recipe1M), and Remote Sensing (using NWPU-Captions and RSICD).

Performance Gains

AdaMLLM consistently outperforms both general-purpose MLLMs and existing domain-specific baselines. In the remote sensing domain, AdaMLLM-8B improved the score on the NWPU benchmark from 26.1 (base LLaVA) to 72.1. In biomedicine, significant gains were observed on benchmarks such as SLAKE and PathVQA.

Radar charts illustrating the performance of AdaMLLM-8B and AdaMLLM-2B against base models across various domain-specific benchmarks.

Across model scales (2B to 11B parameters), the AdaMLLM approach yielded an average improvement of up to +12.9 points over state-of-the-art open-source models.

Discussion and Conclusion

The success of AdaMLLM highlights the importance of task complexity and training efficiency in domain adaptation. While synthetic data may occasionally contain hallucinations, the increased complexity and diversity of these tasks allow models to generalize better than those trained on simple, rule-based data. Future work will focus on refining filtering mechanisms and exploring domain-specific preference optimization to further reduce hallucinations and improve local detail recognition in sensitive fields like medical imaging.

Summary updated

Abstract

Adapting general multimodal large language models (MLLMs) to specific domains, such as scientific and industrial fields, is highly significant in promoting their practical applications. This paper systematically investigates domain adaptation of MLLMs via post-training, focusing on data synthesis, training pipeline, and task evaluation. (1) Data Synthesis: Using only open-source models, we develop a generate-then-filter pipeline that curates diverse visual instruction tasks based on domain-specific image-caption pairs. The resulting data surpass the data synthesized by manual rules or strong closed-source models in enhancing domain-specific performance. (2) Training Pipeline: Unlike general MLLMs that typically adopt a two-stage training paradigm, we find that a single-stage approach is more effective for domain adaptation. (3) Task Evaluation: We conduct extensive experiments in high-impact domains such as biomedicine, food, and remote sensing, by post-training a variety of MLLMs and then evaluating MLLM performance on various domain-specific tasks. Finally, we fully open-source our models, code, and data to encourage future research in this area.
BibTeX
@misc{cheng2024domain,
  author = {Daixuan Cheng and Shaohan Huang and Ziyu Zhu and Xintong Zhang and Xin Zhao and Zhongzhi Luan and Bo Dai and Zhenliang Zhang},
  title = {{On Domain-Adaptive Post-Training for Multimodal Large Language Models}},
  year = {2024},
  eprint = {2411.19930},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.19930},
}
Figure