Improving Image Captioning with Conditional Generative Adversarial Nets

Chen Chen, Shuai Mu, Wanpeng Xiao, Zexiong Ye, Liesi Wu, Qi JuAAAI 2019 · DOI

Introduction

Generating natural language descriptions for images—image captioning—is a cornerstone task in multimodal artificial intelligence. Traditionally, these models are trained using Maximum Likelihood Estimation (MLE). However, MLE suffers from exposure bias, where the model is never exposed to its own errors during training, leading to compounding mistakes at test time. To mitigate this, researchers often turn to Reinforcement Learning (RL) to optimize for language metrics like CIDEr or BLEU.

While RL improves scores, these metrics are imperfect proxies for human language. Models can "game" these scores by over-optimizing for specific keywords, resulting in repetitive or "robotic" phrasing. This paper proposes a framework that combines RL with Conditional Generative Adversarial Nets (cGANs) to produce captions that are both accurate and natural.

The cGAN Framework for Captioning

The core of the proposed method is a tandem architecture consisting of a Generator ($G$) and a Discriminator ($D$).

  1. The Generator ($G$): Utilizing a standard encoder-decoder (CNN-RNN) architecture with attention, the generator produces word sequences based on input images.
  2. The Discriminator ($D$): The authors evaluate two architectures—CNN-based and RNN-based. The discriminator's role is to distinguish between human-written ground truth and machine-generated captions, providing a probability $p$ of "human-likeness."
  3. The Hybrid Reward: To balance descriptive fidelity with naturalness, the model employs a weighted reward function during RL training: $$r = \lambda \cdot p + (1 - \lambda) \cdot s$$ where $s$ represents a standard language metric (e.g., CIDEr) and $\lambda$ is a balancing hyperparameter.
The proposed cGAN framework for image captioning, illustrating the interaction between the Generator, Discriminator, and the hybrid reward mechanism.

The training process utilizes Self-Critical Sequence Training (SCST). By alternating training between the generator and discriminator, the generator learns to produce captions that not only achieve high metric scores but also fool the discriminator by mimicking human linguistic patterns.

Experimental Setup

The framework was evaluated using the MSCOCO dataset, following the standard "Karpathy splits" (113k training, 5k validation, and 5k test images). Performance was measured across five metrics: BLEU-4, METEOR, ROUGE-L, CIDEr, and SPICE.

Results and Discussion

The GAN-augmented models demonstrated consistent improvements across all base architectures. Notably, when applied to the state-of-the-art "Up-Down" attention model, the CIDEr score rose from 120.1 to 125.9. Unlike traditional RL, which often sees one metric improve at the expense of another (e.g., BLEU vs. SPICE), the cGAN approach yielded holistic improvements across the board.

Qualitative comparison of generated captions across different models, highlighting the reduction in repetitive phrasing and improved naturalness in GAN-based approaches.

Qualitative analysis reveals that GAN-trained models are significantly less prone to "stuttering" or redundant phrasing. For instance, where a standard model might output "a group of zebras and a zebra," the GAN-based model produces more concise and natural descriptions like "a group of zebras and other animals."

Challenges and Limitations

Despite its success, the adversarial approach introduces complexities:

  • Stability: The model requires extensive pre-training using MLE before the adversarial phase can begin.
  • Sensitivity: The balancing factor $\lambda$ is critical; over-emphasizing the discriminator reward can lead the model to prioritize style over the actual content of the image.
  • Computational Overhead: Training a discriminator alongside the generator increases memory and processing requirements.

Conclusion

By integrating a learned discriminator into the reinforcement learning loop, this work effectively bridges the gap between mathematical metric optimization and human-like language generation. The resulting models not only achieve higher scores on standard benchmarks but also produce descriptions that are qualitatively superior and more natural to human readers.

Summary updated

BibTeX
@inproceedings{chen2019improving,
  author = {Chen Chen and Shuai Mu and Wanpeng Xiao and Zexiong Ye and Liesi Wu and Qi Ju},
  title = {{Improving Image Captioning with Conditional Generative Adversarial Nets}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.1609/aaai.v33i01.33018142},
}
Figure