Introduction
General-purpose Multimodal Large Language Models (MLLMs), such as LLaVA and Qwen-VL, have demonstrated remarkable versatility in common visual-linguistic tasks. However, their performance often degrades significantly when applied to specialized domains like biomedicine, food science, or remote sensing. Adapting these models to niche fields is traditionally hindered by data scarcity, the high cost of expert annotation, and privacy concerns associated with closed-source synthesis APIs. Furthermore, standard multi-stage training pipelines often suffer from "catastrophic forgetting," where the model loses foundational domain knowledge during the transition from captioning to instruction-following tasks.
Methodology: The AdaMLLM Framework
To address these challenges, we introduce AdaMLLM, a systematic approach for domain-adaptive post-training. The framework consists of two primary components: a "generate-then-filter" data synthesis pipeline and a streamlined single-stage training process.
Visual Instruction Synthesizer
Rather than relying on manual rules or proprietary models, AdaMLLM fine-tunes an open-source MLLM to serve as a Visual Instruction Synthesizer. This synthesizer transforms domain-specific image-caption pairs into high-quality "task triplets" comprising a complex instruction, an informative reasoning-based response, and a precise final answer. To ensure data quality, a consistency-based filter is employed; an LLM verifies the logical alignment between the informative response and the precise answer, discarding contradictory samples.
Single-Stage Post-Training
Unlike the conventional two-stage approach (pre-training on captions followed by instruction tuning), AdaMLLM utilizes a single-stage post-training strategy. By mixing image-captioning tasks with synthetic instruction-following tasks throughout the training process, the model maintains task diversity and prevents over-specialization. This approach ensures that the model retains the ability to generate detailed descriptions while mastering complex domain-specific reasoning.
Experimental Setup and Results
The AdaMLLM framework was evaluated across three high-impact domains: Biomedicine (using PubMed Central data), Food (using Recipe1M), and Remote Sensing (using NWPU-Captions and RSICD).
Performance Gains
AdaMLLM consistently outperforms both general-purpose MLLMs and existing domain-specific baselines. In the remote sensing domain, AdaMLLM-8B improved the score on the NWPU benchmark from 26.1 (base LLaVA) to 72.1. In biomedicine, significant gains were observed on benchmarks such as SLAKE and PathVQA.
Across model scales (2B to 11B parameters), the AdaMLLM approach yielded an average improvement of up to +12.9 points over state-of-the-art open-source models.
Discussion and Conclusion
The success of AdaMLLM highlights the importance of task complexity and training efficiency in domain adaptation. While synthetic data may occasionally contain hallucinations, the increased complexity and diversity of these tasks allow models to generalize better than those trained on simple, rule-based data. Future work will focus on refining filtering mechanisms and exploring domain-specific preference optimization to further reduce hallucinations and improve local detail recognition in sensitive fields like medical imaging.
Summary updated