VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning

Jun Chen, Han Guo, Kai Yi, Boyang Li, Mohamed Elhoseiny

Abstract

The ability to quickly learn from a small quantity of training data widens the range of machine learning applications. We propose a data-efficient image captioning model, VisualGPT, which leverages the linguistic knowledge from a large pretrained language model (LM). A crucial challenge is to balance between the use of visual information in the image and prior linguistic knowledge acquired from pretraining. We designed a novel self-resurrecting encoder-decoder attention mechanism to quickly adapt the pretrained LM as the language decoder on a small amount of in-domain training data. The proposed self-resurrecting activation unit (SRAU) produces sparse activations but has reduced susceptibility to zero gradients. We train VisualGPT on 0.1%, 0.5% and 1% of MSCOCO and Conceptual Captions training data. Under these conditions, we outperform the best baseline model by up to 10.8% CIDEr on MS COCO and up to 5.4% CIDEr on Conceptual Captions. Further, VisualGPT achieves state-of-the-art results on IU X-ray, a medical report generation dataset.

Introduction: The Challenge of Data Scarcity

Deep learning models for image captioning are notoriously data-hungry, typically requiring hundreds of thousands of image-text pairs to perform well. However, in specialized fields—such as medical imaging or low-resource languages—collecting such massive datasets is often impossible or prohibitively expensive.

The challenge isn't just "using a pretrained model." When adapting a massive Pretrained Language Model (PLM) like GPT-2 to a vision task, two primary issues arise:

  1. The Modality Gap: PLMs are trained on general text (like Wikipedia), which differs significantly from image captions that are heavy on nouns and static descriptions.
  2. Knowledge Overwriting: During training, models often "forget" their rich linguistic priors as they struggle to incorporate visual data, a problem exacerbated by standard gating mechanisms that can get stuck with zero gradients.

Methods: Bridging Vision and Language with SRAU

VisualGPT uses an encoder-decoder structure where the decoder is initialized with GPT-2 weights. The core innovation is the Self-Resurrecting Activation Unit (SRAU) within the encoder-decoder attention layer.

Comparison of VisualGPT architecture with standard Transformer variants, showing the integration of pretrained LM weights and the SRAU mechanism.

The SRAU Mechanism

Standard gates often use a sigmoid function to decide how much visual vs. linguistic information to let through. If a gate's output hits zero, the gradient vanishes, and that gate can never "turn back on." The SRAU uses a pair of complementary, asymmetric gates ($B^{vis}$ and $B^{lan}$). These gates produce sparse activations, preventing the model from overwriting its pretrained linguistic knowledge with noisy visual signals. Crucially, because the gates are mathematically linked, even if one gate outputs zero, the gradient from the other gate allows the "dead" gate to be "resurrected" and updated during backpropagation.

Multi-level Fusion

The model also adopts "Meshed Connections," connecting multiple encoder layers to the decoder to provide a richer hierarchy of visual features, ensuring the decoder has access to both low-level and high-level visual information.

Performance in Low-Data Regimes

The authors evaluated VisualGPT on three primary benchmarks, focusing specifically on "low-data" regimes: MS COCO, Conceptual Captions (CC), and IU X-ray (a medical dataset).

Performance comparison of VisualGPT against baselines on COCO and Conceptual Captions datasets across 0.1%, 0.5%, and 1% data splits.

VisualGPT shows a massive performance boost when data is limited. When trained on just 0.1% of the data, VisualGPT outperformed the best baselines by 10.0% CIDEr on MS COCO and 17.9% CIDEr on Conceptual Captions (relative improvement). It also achieved state-of-the-art results on the IU X-ray dataset, proving its utility in specialized, high-stakes domains where data is naturally scarce.

Key Insights

  • GPT > BERT for Captioning: Autoregressive models like GPT are superior to masked language models like BERT for captioning because their training objective matches the sequential nature of generating a caption.
  • The "Noun Bias": MS COCO uses 75% more nouns but 14% fewer verbs than WikiText-2. This justifies the need for a mechanism like SRAU that carefully balances linguistic priors with visual inputs.
  • Layer-wise Specialization: Analysis showed that lower decoder layers focus on syntax, while middle layers handle the fusion of vision and language. The SRAU allows the model to dynamically adjust this balance word-by-word.

Limitations and Future Work

As the amount of in-domain training data increases (e.g., toward 100% of MS COCO), the performance gap between VisualGPT and standard models narrows. When data is abundant, the "prior knowledge" from GPT-2 becomes less critical. Future work could involve exploring how to adapt even larger language models or applying this "self-resurrecting" logic to other multimodal tasks like Visual Question Answering (VQA).

Summary updated

Abstract

The ability to quickly learn from a small quantity oftraining data widens the range of machine learning applications. In this paper, we propose a data-efficient image captioning model, VisualGPT, which leverages the linguistic knowledge from a large pretrained language model(LM). A crucial challenge is to balance between the use of visual information in the image and prior linguistic knowledge acquired from pretraining. We designed a novel self-resurrecting encoder-decoder attention mechanism to quickly adapt the pretrained LM as the language decoder ona small amount of in-domain training data. The proposed self-resurrecting activation unit produces sparse activations but has reduced susceptibility to zero gradients. We train the proposed model, VisualGPT, on 0.1%, 0.5% and 1% of MSCOCO and Conceptual Captions training data. Under these conditions, we outperform the best baseline model by up to 10.8% CIDEr on MS COCO and upto 5.4% CIDEr on Conceptual Captions. Further, Visual-GPT achieves the state-of-the-art result on IU X-ray, a medical report generation dataset. To the best of our knowledge, this is the first work that improves data efficiency of image captioning by utilizing LM pretrained on unimodal data. Our code is available at: https://github.com/Vision-CAIR/VisualGPT.
BibTeX
@inproceedings{chen2021visualgpt,
  author = {Jun Chen and Han Guo and Kai Yi and Boyang Li and Mohamed Elhoseiny},
  title = {{VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
}
Figure