VisualGPT: Data-efficient Image Captioning by Balancing Visual Input and Linguistic Knowledge from Pretraining

Jun Chen, Han Guo, Kai Yi, Boyang Li, Mohamed Elhoseiny

Abstract

The ability to quickly learn from a small quantity of training data widens the range of machine learning applications. In this paper, we propose a data-efficient image captioning model, VisualGPT, which leverages the linguistic knowledge from a large pretrained language model (LM). A crucial challenge is to balance between the use of visual information in the image and prior linguistic knowledge acquired from pretraining. We designed a novel self-resurrecting encoder-decoder attention mechanism to quickly adapt the pretrained LM as the language decoder on a small amount of in-domain training data. The proposed self-resurrecting activation unit produces sparse activations but has reduced susceptibility to zero gradients. We train the proposed model, VisualGPT, on 0.1%, 0.5% and 1% of MSCOCO and Conceptual Captions training data. Under these conditions, we outperform the best baseline model by up to 10.8% CIDEr on MS COCO and up to 5.4% CIDEr on Conceptual Captions. Further, VisualGPT achieves the state-of-the-art result on IU X-ray, a medical report generation dataset.

Introduction

Modern image captioning systems are notoriously data-hungry, often requiring hundreds of thousands of image-text pairs to perform well. However, in specialized domains—such as medical imaging or low-resource languages—collecting high-quality, annotated data is incredibly expensive and slow. While Pretrained Language Models (PLMs) like GPT-2 possess significant linguistic knowledge, adapting them to "see" is difficult. When trained on small datasets, these models often suffer from linguistic overwriting (forgetting general knowledge) or visual neglect (ignoring the image in favor of linguistic priors), leading to hallucinations.

Methodology

VisualGPT addresses these issues by using a pretrained GPT-2 as the decoder and a randomly initialized Transformer as the image encoder. The core innovation is the Self-Resurrecting Activation Unit (SRAU), integrated into a novel encoder-decoder attention mechanism.

Figure 1: The VisualGPT architecture featuring the self-resurrecting encoder-decoder attention mechanism and the integration of pretrained LM weights.

Standard gating mechanisms often suffer from the "vanishing gradient" problem; if a gate's output becomes zero, the model can never "learn" to open it again. The SRAU solves this through:

  • Asymmetric Gating: It utilizes two complementary gates ($B^{vis}$ for vision and $B^{lan}$ for language).
  • Sparse Activations: A threshold ($\tau$) forces small activations to zero, preventing the model from overwriting pretrained weights with noisy, low-value updates from small datasets.
  • Self-Resurrection: Because the gates are asymmetric, even if one gate is zeroed out, gradients can still flow through the other, allowing the model to re-activate previously closed gates as training progresses.

Key Insights: The Noun-Verb Gap

A key hurdle in moving from general NLP to computer vision is the "static object bias." The authors observed that MS COCO uses 75% more nouns but 14% fewer verbs than general text (like WikiText-2). VisualGPT manages this by learning to attend more to visual features for "content words" (nouns like "dog" or "truck") while relying on its internal language model for "functional words" (like "the," "is," or "on").

Experimental Results

VisualGPT was evaluated on 0.1%, 0.5%, and 1% subsets of MS COCO and Conceptual Captions, as well as the IU X-ray medical dataset.

Figure 5: Quantitative performance of VisualGPT compared to state-of-the-art baselines across various low-data training regimes.

VisualGPT shines when data is most scarce:

  • Massive Gains on Small Data: When trained on just 0.1% of MS COCO, VisualGPT outperformed the best baseline by 10.0% CIDEr. On 0.1% of Conceptual Captions, the lead reached 17.9%.
  • Medical SOTA: It achieved state-of-the-art results on the IU X-ray dataset, proving its utility for specialized, high-stakes domains where data is naturally limited.
  • Human Preference: In human studies, participants preferred VisualGPT's captions over baselines, noting it was significantly less likely to hallucinate objects not present in the image.

Conclusion

VisualGPT demonstrates that leveraging pretrained linguistic knowledge is a powerful strategy for data-efficient image captioning. By using sparse, self-resurrecting gates, the model preserves its "common sense" language knowledge while effectively incorporating new visual information. While the performance gap narrows as training data increases, VisualGPT's ability to provide a "cold start" makes it an invaluable tool for specialized and low-resource applications.

Summary updated

Abstract

The ability to quickly learn from a small quantity oftraining data widens the range of machine learning applications. In this paper, we propose a data-efficient image captioning model, VisualGPT, which leverages the linguistic knowledge from a large pretrained language model(LM). A crucial challenge is to balance between the use of visual information in the image and prior linguistic knowledge acquired from pretraining. We designed a novel self-resurrecting encoder-decoder attention mechanism to quickly adapt the pretrained LM as the language decoder ona small amount of in-domain training data. The proposed self-resurrecting activation unit produces sparse activations but has reduced susceptibility to zero gradients. We train the proposed model, VisualGPT, on 0.1%, 0.5% and 1% of MSCOCO and Conceptual Captions training data. Under these conditions, we outperform the best baseline model by up to 10.8% CIDEr on MS COCO and upto 5.4% CIDEr on Conceptual Captions. Further, Visual-GPT achieves the state-of-the-art result on IU X-ray, a medical report generation dataset. To the best of our knowledge, this is the first work that improves data efficiency of image captioning by utilizing LM pretrained on unimodal data. Our code is available at: https://github.com/Vision-CAIR/VisualGPT.
BibTeX
@inproceedings{chen2021visualgpt,
  author = {Jun Chen and Han Guo and Kai Yi and Boyang Li and Mohamed Elhoseiny},
  title = {{VisualGPT: Data-efficient Image Captioning by Balancing Visual Input and Linguistic Knowledge from Pretraining}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
}
Figure