A Semi-supervised Framework for Image Captioning

Wenhu Chen, Aurélien Lucchi, Thomas Hofmann

Abstract State-of-the-art approaches for image captioning require supervised training data consisting of captions with paired image data. These methods are typically unable to use unsupervised data such as textual data with no corresponding images, which is a much more abundant commodity. We here propose a novel way of using such textual data by artificially generating missing visual information. We evaluate this learning approach on a newly designed model that detects visual concepts present in an image and feed them to a reviewer-decoder architecture with an attention mechanism. Unlike previous approaches that encode visual concepts using word embeddings, we instead suggest using regional image features which capture more intrinsic information. The main benefit of this architecture is that it synthesizes meaningful thought vectors that capture salient image properties and then applies a soft attentive decoder to decode the thought vectors and generate image captions. We evaluate our model on both Microsoft COCO and Flickr30K datasets and demonstrate that this model combined with our semi-supervised learning method can largely improve performance and help the model to generate more accurate and diverse captions.

Introduction

The field of image captioning has seen significant progress through deep learning, yet it remains constrained by the availability of high-quality, paired image-text datasets. Creating such datasets, like MS-COCO, is a labor-intensive process requiring tens of thousands of human-annotation hours. In contrast, unpaired textual data is an "abundant commodity" across the web. This paper introduces a semi-supervised framework designed to leverage this vast unsupervised resource by "faking" visual information for text-only corpora, thereby pre-training the majority of the model's parameters.

The Proposed Architecture

The core of the system is a multi-stage pipeline designed to translate visual signals into coherent natural language. As shown in Figure 1, the architecture comprises four primary components:

  1. Visual Concept Detector: A Convolutional Neural Network (CNN) identifies specific "visual concepts" (e.g., "people," "bikes," "street") within localized regions of the input image.
  2. Regional Features: Rather than relying on abstract word embeddings for these concepts, the model extracts high-dimensional CNN feature vectors directly from the identified image regions. These regional features capture intrinsic visual details that simple labels omit.
  3. Input Reviewer: An attentive Long Short-Term Memory (LSTM) network "reviews" the set of regional features to synthesize "thought vectors." This stage acts as a filter, resolving potential detector errors and establishing a global context for the scene.
  4. Soft Attentive Decoder: A final LSTM decodes these thought vectors into a natural language caption, employing an attention mechanism to focus on relevant visual cues at each step of word generation.
The proposed image captioning architecture featuring a Visual Concept Detector, Input Reviewer, and Soft Attentive Decoder.

Semi-supervised Learning through Visual Synthesis

The primary innovation of this work is the ability to train on text-only data. To bridge the gap between text and vision, the authors propose a "fake it 'til you make it" strategy. For a given sentence in an unsupervised corpus, the model extracts key concepts. It then synthesizes "faked" regional features by averaging the feature vectors associated with those concepts in the supervised dataset.

As illustrated in Figure 5, this process allows the model to simulate the visual input it would expect if an image were present. By training the Reviewer and Decoder to reconstruct the original sentence from these synthesized features, the model learns the underlying logic of descriptive language and expands its vocabulary far beyond the limits of paired datasets.

The semi-supervised training pipeline where visual features are synthesized from text-only data to pre-train the model.

Experimental Results

The framework was evaluated on the MS-COCO and Flickr30K datasets. The results demonstrate that semi-supervised pre-training significantly boosts performance across all standard metrics. On MS-COCO, the ensemble model achieved a CIDEr score of 98.2, a substantial improvement over baseline models like Neuraltalk2 (66.0).

The impact was even more pronounced on the smaller Flickr30K dataset, where the influx of knowledge from unsupervised text helped mitigate the scarcity of paired data. Furthermore, the model exhibited greater linguistic diversity, avoiding the repetitive "safe" captions often generated by purely supervised systems.

Conclusion

By treating unpaired text as a source of "missing" visual data, this semi-supervised framework effectively utilizes abundant unsupervised resources to improve image captioning. The combination of regional visual features and a reviewer-decoder architecture provides a robust foundation for generating accurate and diverse descriptions. Future work may extend this approach to specialized domains, such as medical imaging, where paired data is exceptionally rare but descriptive text is plentiful.

Summary updated

Abstract

State-of-the-art approaches for image captioning require supervised training data consisting of captions with paired image data. These methods are typically unable to use unsupervised data such as textual data with no corresponding images, which is a much more abundant commodity. We here propose a novel way of using such textual data by artificially generating missing visual information. We evaluate this learning approach on a newly designed model that detects visual concepts present in an image and feed them to a reviewer-decoder architecture with an attention mechanism. Unlike previous approaches that encode visual concepts using word embeddings, we instead suggest using regional image features which capture more intrinsic information. The main benefit of this architecture is that it synthesizes meaningful thought vectors that capture salient image properties and then applies a soft attentive decoder to decode the thought vectors and generate image captions. We evaluate our model on both Microsoft COCO and Flickr30K datasets and demonstrate that this model combined with our semi-supervised learning method can largely improve performance and help the model to generate more accurate and diverse captions.
BibTeX
@misc{chen2016semi,
  author = {Wenhu Chen and Aurélien Lucchi and Thomas Hofmann},
  title = {{A Semi-supervised Framework for Image Captioning}},
  year = {2016},
  eprint = {1611.05321},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1611.05321},
}
Figure