Image Captioning with Very Scarce Supervised Data: Adversarial Semi-Supervised Learning Approach

Dong-Jin Kim, Jinsoo Choi, Tae-Hyun Oh, In So Kweon

Abstract

Dong-Jin Kim, Jinsoo Choi, Tae-Hyun Oh, In So Kweon. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).

2019.

Introduction

Training robust image captioning models typically necessitates massive datasets of images paired with human-written descriptions. For instance, standard benchmarks like MS COCO rely on over a million human-annotated captions, which are expensive and time-consuming to produce. While unpaired images and descriptive sentences are abundant on the web, leveraging them effectively remains a challenge due to the "modal gap"—the fundamental difference between visual pixel data and linguistic word data.

This paper introduces a semi-supervised framework that achieves high performance using as little as 1% of the paired data typically required. By employing Generative Adversarial Networks (GANs), the model "pseudo-labels" and learns from vast amounts of unpaired images and text.

Data Scarcity and the Semi-Supervised Setup

The core challenge addressed is the reliance on paired data ($D_p$). The authors propose using a small "seed" of paired data to guide learning on much larger, unpaired image ($D_u^x$) and text ($D_u^y$) datasets.

The proposed framework leverages a small amount of paired data alongside large volumes of unpaired image and text data.

Methodology: Adversarial Semi-Supervised Learning

The architecture bridges the modal gap through several key components:

  1. Shared Feature Space: A CNN (e.g., ResNet) encodes images and an LSTM encodes captions into a common mathematical space.
  2. The Discriminator ($D$): Trained to distinguish between correctly matched image-caption pairs and mismatched or synthetic ones.
  3. Pseudo-Label Search: For an unpaired image, the model searches a pool of unpaired captions. The Discriminator scores potential pairs, selecting the best match as a "pseudo-label."
  4. Confidence Weighting: To handle noise in pseudo-labels, the model assigns a confidence score based on the discriminator's output. High-confidence pairs have a greater influence on training.
  5. Feature Transformers: Small networks ($T_{v \to c}$ and $T_{c \to v}$) map visual features to caption features and vice versa, helping the model "imagine" cross-modal correspondences.

Experimental Setup

The authors evaluated their approach using:

  • MS COCO: The standard benchmark.
  • Scarcely-paired COCO: A custom setup using only 1,133 paired samples (1% of the total).
  • Unlabeled-COCO: 123k images without captions.
  • Shutterstock: 2.2 million unpaired sentences to test real-world, noisy data performance.

Key Results

The proposed method demonstrates significant efficiency gains:

  • Quantitative Performance: In the 1% paired data setup, the method achieved a CIDEr score of 55.2, vastly outperforming the "Paired Only" baseline (36.0) and modified CycleGAN approaches (37.7).
  • Data Efficiency: The model trained on 1% paired data plus unpaired data performed similarly to a standard model trained on 60% paired data, representing a massive reduction in required human labor.
  • Vocabulary Expansion: The model learned "novel" words (e.g., "growling," "herded") that appeared only in the unpaired text, not in the 1% seed data.

Limitations and Future Work

While effective, the approach faces computational challenges in searching for pseudo-labels across massive datasets. Currently, the authors search a subset per minibatch, but scaling to billions of sentences would require approximate nearest neighbor techniques. Additionally, the model remains sensitive to the representativeness of the 1% seed data. Future work may extend this framework to video captioning or visual question answering (VQA).

Summary updated

BibTeX
@inproceedings{kim2019image,
  author = {Dong-Jin Kim and Jinsoo Choi and Tae-Hyun Oh and In So Kweon},
  title = {{Image Captioning with Very Scarce Supervised Data: Adversarial Semi-Supervised Learning Approach}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=aUSvhDDDyjC},
}
Figure