Abstract
Dong-Jin Kim, Jinsoo Choi, Tae-Hyun Oh, In So Kweon. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
2019.
Introduction
Training robust image captioning models typically necessitates massive datasets of images paired with human-written descriptions. For instance, standard benchmarks like MS COCO rely on over a million human-annotated captions, which are expensive and time-consuming to produce. While unpaired images and descriptive sentences are abundant on the web, leveraging them effectively remains a challenge due to the "modal gap"—the fundamental difference between visual pixel data and linguistic word data.
This paper introduces a semi-supervised framework that achieves high performance using as little as 1% of the paired data typically required. By employing Generative Adversarial Networks (GANs), the model "pseudo-labels" and learns from vast amounts of unpaired images and text.
Data Scarcity and the Semi-Supervised Setup
The core challenge addressed is the reliance on paired data ($D_p$). The authors propose using a small "seed" of paired data to guide learning on much larger, unpaired image ($D_u^x$) and text ($D_u^y$) datasets.
Methodology: Adversarial Semi-Supervised Learning
The architecture bridges the modal gap through several key components:
- Shared Feature Space: A CNN (e.g., ResNet) encodes images and an LSTM encodes captions into a common mathematical space.
- The Discriminator ($D$): Trained to distinguish between correctly matched image-caption pairs and mismatched or synthetic ones.
- Pseudo-Label Search: For an unpaired image, the model searches a pool of unpaired captions. The Discriminator scores potential pairs, selecting the best match as a "pseudo-label."
- Confidence Weighting: To handle noise in pseudo-labels, the model assigns a confidence score based on the discriminator's output. High-confidence pairs have a greater influence on training.
- Feature Transformers: Small networks ($T_{v \to c}$ and $T_{c \to v}$) map visual features to caption features and vice versa, helping the model "imagine" cross-modal correspondences.
Experimental Setup
The authors evaluated their approach using:
- MS COCO: The standard benchmark.
- Scarcely-paired COCO: A custom setup using only 1,133 paired samples (1% of the total).
- Unlabeled-COCO: 123k images without captions.
- Shutterstock: 2.2 million unpaired sentences to test real-world, noisy data performance.
Key Results
The proposed method demonstrates significant efficiency gains:
- Quantitative Performance: In the 1% paired data setup, the method achieved a CIDEr score of 55.2, vastly outperforming the "Paired Only" baseline (36.0) and modified CycleGAN approaches (37.7).
- Data Efficiency: The model trained on 1% paired data plus unpaired data performed similarly to a standard model trained on 60% paired data, representing a massive reduction in required human labor.
- Vocabulary Expansion: The model learned "novel" words (e.g., "growling," "herded") that appeared only in the unpaired text, not in the 1% seed data.
Limitations and Future Work
While effective, the approach faces computational challenges in searching for pseudo-labels across massive datasets. Currently, the authors search a subset per minibatch, but scaling to billions of sentences would require approximate nearest neighbor techniques. Additionally, the model remains sensitive to the representativeness of the 1% seed data. Future work may extend this framework to video captioning or visual question answering (VQA).
Summary updated