TL;DR
MoColl (Model Collaboration) is a novel framework that addresses the "specialization vs. generalization" trade-off in image captioning. By pairing a general-purpose LLM agent with a domain-specific visual tool, the system decomposes complex captioning into an iterative "interview." This approach achieves state-of-the-art results in technical fields like radiology report generation.
Abstract
Image captioning in specialized domains, such as medicine, requires both deep domain expertise and broad contextual reasoning. Existing models often struggle with this balance, either lacking niche precision or suffering from "catastrophic forgetting" when fine-tuned. MoColl introduces a collaborative loop where a Large Language Model (LLM) agent interacts with a specialized Visual Question Answering (VQA) model. Through an iterative questioning process and an agent-guided tuning strategy that filters synthetic training data, MoColl demonstrates superior performance and robustness across standard and cross-dataset benchmarks.
Problem Statement
In specialized fields like radiology, image captioning is a high-stakes task. Researchers currently face two primary limitations:
- Specialized Models: These models excel at identifying domain-specific features (e.g., lung nodules) but lack the linguistic flexibility to produce coherent, nuanced reports.
- General Vision-Language Models (VLMs): While articulate and capable of broad reasoning, these models often struggle with niche technical details. Heavy fine-tuning on domain-specific data frequently leads to "catastrophic forgetting," where the model loses its general reasoning capabilities.
MoColl aims to bridge this gap by maintaining the "brain" of a general LLM while equipping it with the "eyes" of a highly trained specialist.
Methodology
The MoColl framework is built on a collaborative loop rather than a single end-to-end model. The process is divided into two primary components: Agent-Based Collaboration and Agent-Guided Tuning.
Agent-Based Collaboration
Instead of a single-pass generation, MoColl utilizes an LLM-based agent to act as the central reasoning unit. The agent initiates an iterative "interview" by asking a series of targeted questions about the image (e.g., "Are the lung fields clear?"). A specialized VQA model then provides technical answers. Finally, the agent synthesizes these Q&A pairs into a final, coherent caption.
Agent-Guided Tuning
To refine the VQA "tool," the authors introduced Agent-Guided Tuning. In this stage, the agent acts as a teacher by generating synthetic Q&A pairs from training data. It then filters these pairs by comparing the generated answers to ground-truth captions, discarding hallucinations and retaining only high-quality data for fine-tuning.
Technically, the VQA model employs a LLaVA-style architecture, combining a CLIP vision encoder, a projection layer, and Llama-3. During tuning, the vision encoder and LLM weights remain frozen; only the projection network and specific adapters are updated to preserve the model's base knowledge.
Experimental Setup and Datasets
The researchers validated MoColl using two gold-standard chest X-ray datasets:
- IU-Xray: 3,955 reports and 7,470 images.
- MIMIC-CXR: 206,000 reports and 473,000 images.
A critical component of the evaluation was cross-dataset generalization—training on MIMIC-CXR and testing on IU-Xray—to ensure the model learned robust reasoning rather than dataset-specific patterns.
Results and Discussion
MoColl consistently outperformed both specialized encoder-decoder models and general VLMs across standard NLP metrics, including BLEU, Meteor, ROUGE-L, and Cider.
- Within-Dataset Performance: On MIMIC-CXR, MoColl achieved a ROUGE-L of 0.2349 and a Cider score of 0.0932, significantly surpassing the baseline Aligned VLM scores of 0.1614 and 0.0480.
- Cross-Dataset Robustness: When tested on IU-Xray after training on MIMIC, MoColl maintained a high ROUGE-L of 0.2822, demonstrating significantly better stability than standard fine-tuned models.
- Qualitative Accuracy: Visual analysis showed that MoColl successfully identified subtle nuances, such as "T-spine osteophytes" and "aorta tortuosity," which general models frequently overlooked.
Key insights from the study suggest that task decomposition is vital for reducing hallucinations. Furthermore, increasing the number of questions in the agent's "interview" directly correlates with higher vocabulary precision (BLEU-1), as more questions yield more accurate technical keywords.
Limitations and Future Work
Despite its success, MoColl faces challenges regarding data requirements and computational overhead. The Agent-Guided Tuning phase requires substantial domain-specific data, which may limit effectiveness in low-resource scenarios like rare disease diagnosis. Additionally, the iterative dialogue process is more computationally expensive than single-pass models.
Future research will focus on more efficient data utilization and expanding the MoColl framework to other multi-modal tasks, such as technical manual analysis and complex scene understanding.
Summary updated