Image Captioning with Very Scarce Supervised Data: Adversarial Semi-Supervised Learning Approach

Dong-Jin Kim, Jinsoo Choi, Tae-Hyun Oh, In So Kweon

Abstract

Constructing an organized dataset comprised of a large number of images and several captions for each image is a laborious task, which requires vast human effort. On the other hand, collecting a large number of images and sentences separately may be immensely easier. In this paper, we develop a novel data-efficient semi-supervised framework for training an image captioning model. We leverage massive unpaired image and caption data by learning to associate them. To this end, our proposed semi-supervised learning method assigns pseudo-labels to unpaired samples via Generative Adversarial Networks to learn the joint distribution of image and caption. To evaluate, we construct scarcely-paired COCO dataset, a modified version of MS COCO caption dataset. The empirical results show the effectiveness of our method compared to several strong baselines, especially when the amount of the paired samples are scarce.

Introduction

Modern image captioning models are typically "data-hungry," requiring massive datasets like MS COCO, which contains over 1.2 million human-annotated captions. Constructing these datasets is a Herculean task involving thousands of hours of manual labor. Conversely, "unpaired" data—billions of images on Flickr or millions of descriptive sentences on the web—is incredibly easy to find but difficult to use because there is no explicit link between a specific image and a specific description. This paper tackles the challenge of data-efficient learning: training a captioning model using a tiny "seed" of paired data and a mountain of unpaired data.

Methodology

The proposed Adversarial Semi-Supervised Learning approach consists of three main components:

  1. Captioning Model (CNN-RNN): A standard encoder-decoder that takes an image and generates text.
  2. Discriminator ($D$): Unlike a standard GAN discriminator, this is trained to distinguish between semantically matching image-caption pairs and mismatched ones.
  3. Pseudo-label Search Module: For every unpaired image, the model searches the pool of unpaired captions to find the one that the discriminator ranks as the "most likely" match. This caption becomes a pseudo-label.

To handle noise, the authors implement Confidence-based Re-weighting. The discriminator’s output score acts as a confidence measure; if the match is weak, the loss from that sample is down-weighted. They also incorporate a triplet loss to ensure that the latent features of images and their corresponding captions are pushed closer together in the embedding space.

Experimental Setup

The researchers evaluated their method on several configurations of the MS COCO dataset, specifically focusing on scenarios where paired data is limited.

Examples of paired data ($D_p$) used for initial training.

The datasets included:

  • Scarcely-paired COCO: A custom benchmark where only 1% of the original pairs (approx. 1,133 images) are kept as "paired," while the rest are treated as unpaired.
  • Partially labeled COCO: Utilizing the official "Unlabeled-COCO" split (123k images).
  • Web-Crawled Data: 2.2 million sentences crawled from Shutterstock as unpaired text data to simulate a real-world "in-the-wild" scenario.

Results and Discussion

On the 1% paired COCO setup, the proposed method achieved a CIDEr score of 55.2, significantly outperforming the "Paired Only" baseline (36.0) and a modified CycleGAN (37.7). The model generated captions comparable in quality to baseline models trained on 10% or more paired data.

A key finding was the model's ability to expand its vocabulary. It learned to use words (like "growling" or "herded") that appeared in the unpaired text pool but were entirely absent from the tiny 1% paired training set. This suggests the model is truly learning semantic concepts. Furthermore, the method is architecture-agnostic, showing consistent improvements when applied to larger backbones like the Up-Down attention model.

Conclusion

This work demonstrates that adversarial semi-supervised learning can significantly reduce the human effort required for image captioning. While searching for pseudo-labels is computationally expensive and a small "seed" of paired data is still required, the ability to leverage massive unpaired datasets opens new avenues for training robust models in data-scarce domains. Future work may explore completely unsupervised initialization to further reduce the reliance on human annotation.

Summary updated

Abstract

Constructing an organized dataset comprised of a large number of images and several captions for each image is a laborious task, which requires vast human effort. On the other hand, collecting a large number of images and sentences separately may be immensely easier. In this paper, we develop a novel data-efficient semi-supervised framework for training an image captioning model. We leverage massive unpaired image and caption data by learning to associate them. To this end, our proposed semi-supervised learning method assigns pseudo-labels to unpaired samples via Generative Adversarial Networks to learn the joint distribution of image and caption. To evaluate, we construct scarcely-paired COCO dataset, a modified version of MS COCO caption dataset. The empirical results show the effectiveness of our method compared to several strong baselines, especially when the amount of the paired samples are scarce.
BibTeX
@inproceedings{kim2019image,
  author = {Dong-Jin Kim and Jinsoo Choi and Tae-Hyun Oh and In So Kweon},
  title = {{Image Captioning with Very Scarce Supervised Data: Adversarial Semi-Supervised Learning Approach}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=aUSvhDDDyjC},
}
Figure