Towards Diverse and Natural Image Descriptions via a Conditional GAN

Bo Dai, Dahua Lin, Raquel Urtasun, Sanja Fidler

Abstract

Traditional image captioning models, primarily trained using Maximum Likelihood Estimation (MLE), often generate "safe" but repetitive and robotic descriptions. This paper proposes a Conditional Generative Adversarial Network (CGAN) framework designed to produce more diverse and natural-sounding captions. By combining an LSTM-based generator with a semantic evaluator and utilizing reinforcement learning with Monte Carlo rollouts, the proposed model (G-GAN) generates descriptions that are significantly more human-like. Human evaluation shows that G-GAN is preferred over MLE-based models in 61% of cases and performs competitively against ground-truth human descriptions.

Introduction

The field of image captioning has seen significant progress, yet most state-of-the-art models rely on Maximum Likelihood Estimation (MLE). While effective for accuracy, MLE encourages models to output the most frequent n-grams found in training data, resulting in rigid and generic descriptions. Furthermore, standard evaluation metrics like BLEU and CIDEr reward exact word-for-word matches, often penalizing more descriptive and creative human-like sentences. This "BLEU trap" creates a gap between high metric scores and actual linguistic naturalness. The authors aim to bridge this gap by prioritizing diversity and vitality in generated text.

Methodology

The authors introduce a Conditional GAN (CGAN) framework consisting of two primary networks: a Generator ($G$) and an Evaluator ($E$).

Generator and Evaluator Architecture

The Generator is an LSTM-based decoder that receives image features from a VGG16 CNN and a latent random vector ($z$). The inclusion of $z$ allows the model to sample from a latent space, enabling the generation of multiple, diverse descriptions for a single image. The Evaluator ($E$) is trained to assess the "naturalness" and semantic relevance of the generated captions, ensuring they accurately reflect the visual content rather than just sounding fluent.

Architectural overview of the Generator (G) and Evaluator (E) for sentence and paragraph generation.

Training via Reinforcement Learning

Because text generation involves discrete sampling, standard backpropagation is inapplicable. The authors employ Policy Gradient methods to optimize the generator. To address the "sparse reward" problem—where feedback is only available at the end of a sentence—they implement Monte Carlo rollouts. This technique simulates multiple potential sentence completions to provide the generator with early, intermediate feedback, stabilizing the training process.

Experimental Results

The model was evaluated on the MSCOCO and Flickr30k datasets. A key finding was the discrepancy between traditional metrics and human judgment. While G-GAN often scored lower on BLEU than G-MLE, human evaluators overwhelmingly preferred its output.

Human Preference Study

In a user study involving 30 evaluators, G-GAN was preferred over the MLE baseline in 61% of comparisons. Notably, G-GAN also proved competitive against actual human descriptions, winning in 24% of head-to-head matchups, a significant improvement over the 9% win rate of the MLE model.

Human preference study results showing G-GAN's superiority over G-MLE and its competitiveness with human descriptions.

Discussion and Conclusion

The results suggest that the Evaluator ($E$-GAN) serves as a more reliable metric for human-like quality than static n-gram matching. However, the model faces limitations; the pursuit of diversity occasionally leads to the inclusion of incorrect specific details (e.g., wrong colors or counts). Future work will focus on improving the capture of fine-grained details while maintaining the descriptive vitality that makes G-GAN's output feel natural. This framework demonstrates that moving beyond MLE is essential for creating AI that communicates with the nuance and variety of human speech.

Summary updated

BibTeX
@inproceedings{dai2017towards,
  author = {Bo Dai and Dahua Lin and Raquel Urtasun and Sanja Fidler},
  title = {{Towards Diverse and Natural Image Descriptions via a Conditional GAN}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2017},
  doi = {10.1109/iccv.2017.323},
}
Figure