MoColl: Agent-Based Specific and General Model Collaboration for Image Captioning

Pu Yang, Bin DongarXiv 2025 · DOI · arXiv

TL;DR

MoColl (Model Collaboration) is a novel framework that addresses the "specialization vs. generalization" trade-off in image captioning. By pairing a general-purpose LLM agent with a domain-specific visual tool, the system decomposes complex captioning into an iterative "interview." This approach achieves state-of-the-art results in technical fields like radiology report generation.

Abstract

Image captioning in specialized domains, such as medicine, requires both deep domain expertise and broad contextual reasoning. Existing models often struggle with this balance, either lacking niche precision or suffering from "catastrophic forgetting" when fine-tuned. MoColl introduces a collaborative loop where a Large Language Model (LLM) agent interacts with a specialized Visual Question Answering (VQA) model. Through an iterative questioning process and an agent-guided tuning strategy that filters synthetic training data, MoColl demonstrates superior performance and robustness across standard and cross-dataset benchmarks.

Problem Statement

In specialized fields like radiology, image captioning is a high-stakes task. Researchers currently face two primary limitations:

  1. Specialized Models: These models excel at identifying domain-specific features (e.g., lung nodules) but lack the linguistic flexibility to produce coherent, nuanced reports.
  2. General Vision-Language Models (VLMs): While articulate and capable of broad reasoning, these models often struggle with niche technical details. Heavy fine-tuning on domain-specific data frequently leads to "catastrophic forgetting," where the model loses its general reasoning capabilities.

MoColl aims to bridge this gap by maintaining the "brain" of a general LLM while equipping it with the "eyes" of a highly trained specialist.

Methodology

The MoColl framework is built on a collaborative loop rather than a single end-to-end model. The process is divided into two primary components: Agent-Based Collaboration and Agent-Guided Tuning.

Agent-Based Collaboration

Instead of a single-pass generation, MoColl utilizes an LLM-based agent to act as the central reasoning unit. The agent initiates an iterative "interview" by asking a series of targeted questions about the image (e.g., "Are the lung fields clear?"). A specialized VQA model then provides technical answers. Finally, the agent synthesizes these Q&A pairs into a final, coherent caption.

The collaborative loop between the LLM agent and the specialized VQA model.

Agent-Guided Tuning

To refine the VQA "tool," the authors introduced Agent-Guided Tuning. In this stage, the agent acts as a teacher by generating synthetic Q&A pairs from training data. It then filters these pairs by comparing the generated answers to ground-truth captions, discarding hallucinations and retaining only high-quality data for fine-tuning.

The two-stage training pipeline: Warm-up alignment followed by Agent-guided tuning.

Technically, the VQA model employs a LLaVA-style architecture, combining a CLIP vision encoder, a projection layer, and Llama-3. During tuning, the vision encoder and LLM weights remain frozen; only the projection network and specific adapters are updated to preserve the model's base knowledge.

Experimental Setup and Datasets

The researchers validated MoColl using two gold-standard chest X-ray datasets:

  • IU-Xray: 3,955 reports and 7,470 images.
  • MIMIC-CXR: 206,000 reports and 473,000 images.

A critical component of the evaluation was cross-dataset generalization—training on MIMIC-CXR and testing on IU-Xray—to ensure the model learned robust reasoning rather than dataset-specific patterns.

Results and Discussion

MoColl consistently outperformed both specialized encoder-decoder models and general VLMs across standard NLP metrics, including BLEU, Meteor, ROUGE-L, and Cider.

  • Within-Dataset Performance: On MIMIC-CXR, MoColl achieved a ROUGE-L of 0.2349 and a Cider score of 0.0932, significantly surpassing the baseline Aligned VLM scores of 0.1614 and 0.0480.
  • Cross-Dataset Robustness: When tested on IU-Xray after training on MIMIC, MoColl maintained a high ROUGE-L of 0.2822, demonstrating significantly better stability than standard fine-tuned models.
  • Qualitative Accuracy: Visual analysis showed that MoColl successfully identified subtle nuances, such as "T-spine osteophytes" and "aorta tortuosity," which general models frequently overlooked.

Key insights from the study suggest that task decomposition is vital for reducing hallucinations. Furthermore, increasing the number of questions in the agent's "interview" directly correlates with higher vocabulary precision (BLEU-1), as more questions yield more accurate technical keywords.

Limitations and Future Work

Despite its success, MoColl faces challenges regarding data requirements and computational overhead. The Agent-Guided Tuning phase requires substantial domain-specific data, which may limit effectiveness in low-resource scenarios like rare disease diagnosis. Additionally, the iterative dialogue process is more computationally expensive than single-pass models.

Future research will focus on more efficient data utilization and expanding the MoColl framework to other multi-modal tasks, such as technical manual analysis and complex scene understanding.

Summary updated

Abstract

Image captioning is a critical task at the intersection of computer vision and natural language processing, with wide-ranging applications across various domains. For complex tasks such as diagnostic report generation, deep learning models require not only domain-specific image-caption datasets but also the incorporation of relevant general knowledge to provide contextual accuracy. Existing approaches exhibit inherent limitations: specialized models excel in capturing domain-specific details but lack generalization, while vision-language models (VLMs) built on large language models (LLMs) leverage general knowledge but struggle with domain-specific adaptation. To address these limitations, this paper proposes a novel agent-enhanced model collaboration framework, which we call MoColl, designed to effectively integrate domain-specific and general knowledge. Specifically, our approach is to decompose complex image captioning tasks into a series of interconnected question-answer subtasks. A trainable visual question answering (VQA) model is employed as a specialized tool to focus on domain-specific visual analysis, answering task-specific questions based on image content. Concurrently, an LLM-based agent with general knowledge formulates these questions and synthesizes the resulting question-answer pairs into coherent captions. Beyond its role in leveraging the VQA model, the agent further guides its training to enhance its domain-specific capabilities. Experimental results on radiology report generation validate the effectiveness of the proposed framework, demonstrating significant improvements in the quality of generated reports.
BibTeX
@misc{yang2025mocoll,
  author = {Pu Yang and Bin Dong},
  title = {{MoColl: Agent-Based Specific and General Model Collaboration for Image Captioning}},
  year = {2025},
  eprint = {2501.01834},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2501.01834},
}
Figure