Abstract
The ability to quickly learn from a small quantity of training data widens the range of machine learning applications. In this paper, we propose a data-efficient image captioning model, VisualGPT, which leverages the linguistic knowledge from a large pretrained language model (LM). A crucial challenge is to balance between the use of visual information in the image and prior linguistic knowledge acquired from pretraining. We designed a novel self-resurrecting encoder-decoder attention mechanism to quickly adapt the pretrained LM as the language decoder on a small amount of in-domain training data. The proposed self-resurrecting activation unit produces sparse activations but has reduced susceptibility to zero gradients. We train the proposed model, VisualGPT, on 0.1%, 0.5% and 1% of MSCOCO and Conceptual Captions training data. Under these conditions, we outperform the best baseline model by up to 10.8% CIDEr on MS COCO and up to 5.4% CIDEr on Conceptual Captions. Further, VisualGPT achieves the state-of-the-art result on IU X-ray, a medical report generation dataset.
Introduction
Modern image captioning systems are notoriously data-hungry, often requiring hundreds of thousands of image-text pairs to perform well. However, in specialized domains—such as medical imaging or low-resource languages—collecting high-quality, annotated data is incredibly expensive and slow. While Pretrained Language Models (PLMs) like GPT-2 possess significant linguistic knowledge, adapting them to "see" is difficult. When trained on small datasets, these models often suffer from linguistic overwriting (forgetting general knowledge) or visual neglect (ignoring the image in favor of linguistic priors), leading to hallucinations.
Methodology
VisualGPT addresses these issues by using a pretrained GPT-2 as the decoder and a randomly initialized Transformer as the image encoder. The core innovation is the Self-Resurrecting Activation Unit (SRAU), integrated into a novel encoder-decoder attention mechanism.
Standard gating mechanisms often suffer from the "vanishing gradient" problem; if a gate's output becomes zero, the model can never "learn" to open it again. The SRAU solves this through:
- Asymmetric Gating: It utilizes two complementary gates ($B^{vis}$ for vision and $B^{lan}$ for language).
- Sparse Activations: A threshold ($\tau$) forces small activations to zero, preventing the model from overwriting pretrained weights with noisy, low-value updates from small datasets.
- Self-Resurrection: Because the gates are asymmetric, even if one gate is zeroed out, gradients can still flow through the other, allowing the model to re-activate previously closed gates as training progresses.
Key Insights: The Noun-Verb Gap
A key hurdle in moving from general NLP to computer vision is the "static object bias." The authors observed that MS COCO uses 75% more nouns but 14% fewer verbs than general text (like WikiText-2). VisualGPT manages this by learning to attend more to visual features for "content words" (nouns like "dog" or "truck") while relying on its internal language model for "functional words" (like "the," "is," or "on").
Experimental Results
VisualGPT was evaluated on 0.1%, 0.5%, and 1% subsets of MS COCO and Conceptual Captions, as well as the IU X-ray medical dataset.
VisualGPT shines when data is most scarce:
- Massive Gains on Small Data: When trained on just 0.1% of MS COCO, VisualGPT outperformed the best baseline by 10.0% CIDEr. On 0.1% of Conceptual Captions, the lead reached 17.9%.
- Medical SOTA: It achieved state-of-the-art results on the IU X-ray dataset, proving its utility for specialized, high-stakes domains where data is naturally limited.
- Human Preference: In human studies, participants preferred VisualGPT's captions over baselines, noting it was significantly less likely to hallucinate objects not present in the image.
Conclusion
VisualGPT demonstrates that leveraging pretrained linguistic knowledge is a powerful strategy for data-efficient image captioning. By using sparse, self-resurrecting gates, the model preserves its "common sense" language knowledge while effectively incorporating new visual information. While the performance gap narrows as training data increases, VisualGPT's ability to provide a "cold start" makes it an invaluable tool for specialized and low-resource applications.
Summary updated