On Domain-Specific Post-Training for Multimodal Large Language Models

Daixuan Cheng, Shaohan Huang, Ziyu Zhu, Xintong Zhang, Wayne Xin Zhao, Zhongzhi Luan, Bo Dai, Zhenliang Zhang

TL;DR

This paper tackles the challenge of adapting general-purpose multimodal large language models (MLLMs) to specialized domains like medicine and food. They introduce a novel "generate-then-filter" pipeline using open-source models for creating high-quality, domain-specific training data and propose a more efficient single-stage training approach, leading to significant performance gains on domain-specific tasks.

What Problem Does This Paper Address?

While general MLLMs are impressive, their performance drops significantly when applied to specialized domains. This is due to a lack of domain-specific training data and the difficulty in creating it. Existing methods often rely on proprietary models (raising privacy concerns) or manual rules, which can be labor-intensive and less effective. Furthermore, the standard two-stage training pipeline for MLLMs, while effective for general tasks, might not be optimal for domain adaptation where task diversity can be limited. This paper aims to bridge this gap by developing a robust and efficient method for domain-specific MLLM adaptation using only open-source tools.

Methods

The authors propose a two-pronged approach:

  1. Domain-Specific Data Synthesis:
    • Visual Instruction Synthesizer: They fine-tune an open-source MLLM to act as a "synthesizer." This model takes domain-specific image-caption pairs and generates diverse visual instruction tasks, including an instruction, an informative response (with reasoning), and a precise response. This approach leverages existing data and avoids the need for manual annotation.
  2. Consistency-Based Filter: To ensure the quality of synthesized data, they employ a filter that checks for consistency between the informative and precise responses. This helps to discard inaccurate or ambiguous outputs.
  3. Modality-Balanced Tuning: To encourage better utilization of both image and text, they replace 10% of training images with blank ones during synthesizer fine-tuning.
  4. Single-Stage Post-Training Pipeline:
    • Instead of the traditional two-stage training (image-caption pairs first, then visual instructions), they propose a single-stage approach. This combines both image-captioning tasks and the synthesized visual instruction tasks into one training phase. The rationale is to maintain task diversity and improve efficiency, especially in domains with limited task variety.

Datasets

The paper utilizes several datasets for training and evaluation:

  • Seed Data for Synthesizer Training: A combination of VisionFLAN and ALLaVA datasets, covering a wide range of image domains and task types.
  • Image-Caption Data Sources for Domain Adaptation:
    • Biomedicine: PMCRaw and PMCRefined (from PubMed Central).
  • Food: Recipe1M (Salvador et al., 2017).
  • Remote Sensing: NWPU-Captions, RSICD, RSITMD, Sydney-Captions, and UCM-Captions.
  • Domain-Specific Evaluation Datasets:
    • Biomedicine: SLAKE, PathVQA, VQA-RAD, and PMC-VQA.
  • Food: Recipe1M (for recipe generation), Nutrition5K (ingredient prediction), Food101 (category classification), and FoodSeg103 (multi-label classification).
  • Remote Sensing: CLRS, UC Merced, FloodNet, and NWPU-Captions.

Key Insights and Ideas

  • Open-Source Power for Domain Adaptation: The paper demonstrates that high-quality domain-specific data can be generated effectively using only open-source MLLMs, overcoming privacy concerns associated with closed-source models.
  • Consistency is Key for Data Quality: The consistency-based filter is crucial for ensuring the accuracy and reliability of synthesized training data, significantly improving downstream performance.
  • Single-Stage Training for Domain Specificity: A single-stage training pipeline is more effective for domain adaptation than the traditional two-stage approach, likely due to better preservation of task diversity and reduced catastrophic forgetting.
  • Modality Balancing Enhances Robustness: Replacing some images with blanks during synthesizer training improves its ability to leverage textual information and makes it more robust to challenging visual inputs.

Key Results

  • Performance Gains: The proposed method, AdaMLLM, consistently outperforms general MLLMs and various baselines across multiple domains (biomedicine, food, remote sensing) and tasks. For instance, in biomedicine, AdaMLLM-8B from PMCRaw achieves scores like 56.8 on SLAKE (OPEN) and 76.4 on PathVQA (OPEN), significantly higher than the base LLaVA-v1.6-8B.
  • Synthesizer Effectiveness: The generated synthetic tasks are shown to be more effective than those created by manual rules, GPT-4, and GPT-4V in enhancing domain-specific performance.
  • Single-Stage Superiority: In most experiments, single-stage training with their synthetic data outperforms two-stage training.
  • Task Diversity and Domain Knowledge: The paper highlights that task diversity and domain knowledge utilization are critical factors for successful domain adaptation.

Limitations and Future Work

  • Hallucinations: The authors acknowledge that synthetic data, even with filtering, can still be prone to hallucinations, a common issue in LLM-generated content. Further improvements are needed to enhance response reliability.
  • Domain-Specific Nuances: Future work could explore tailoring approaches to specific domain preferences, such as focusing on semantic-level information for animal domains versus local detail for medical images.
  • Open-Sourcing: The authors plan to open-source their models, code, and data to facilitate further research in MLLM domain adaptation.

Summary updated

Abstract

Recent years have witnessed the rapid development of general multimodal large language models (MLLMs). However, adapting general MLLMs to specific domains, such as scientific fields and industrial applications, remains less explored. This paper systematically investigates domain adaptation of MLLMs through post-training, focusing on data synthesis, training pipelines, and task evaluation. (1) Data Synthesis: Using open-source models, we develop a visual instruction synthesizer that effectively generates diverse visual instruction tasks from domain-specific image-caption pairs. Our synthetic tasks surpass those generated by manual rules, GPT-4, and GPT-4V in enhancing the domain-specific performance of MLLMs. (2) Training Pipeline: While the two-stage training--initially on image-caption pairs followed by visual instruction tasks--is commonly adopted for developing general MLLMs, we apply a single-stage training pipeline to enhance task diversity for domain-specific post-training. (3) Task Evaluation: We conduct experiments in two domains, biomedicine and food, by post-training MLLMs of different sources and scales (e.g., Qwen2-VL-2B, LLaVA-v1.6-8B, Llama-3.2-11B), and then evaluating MLLM performance on various domain-specific tasks. To support further research in MLLM domain adaptation, we will open-source our implementations.
BibTeX
@misc{cheng2024domain,
  author = {Daixuan Cheng and Shaohan Huang and Ziyu Zhu and Xintong Zhang and Wayne Xin Zhao and Zhongzhi Luan and Bo Dai and Zhenliang Zhang},
  title = {{On Domain-Specific Post-Training for Multimodal Large Language Models}},
  year = {2024},
  eprint = {2411.19930},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.19930},
}
Figure