Interactive Dual Generative Adversarial Networks for Image Captioning

Junhao Liu, Kai Wang, Chunpu Xu, Zhou Zhao, Ruifeng Xu, Ying Shen, Min Yang

Abstract

Image captioning requires a delicate balance between visual grounding and linguistic fluency. Traditional methods often struggle to bridge the gap between flexible generation-based approaches and grammatically robust retrieval-based methods. This paper introduces the Interactive Dual Generative Adversarial Network (IDGAN), a framework that facilitates mutual feedback between a generation-based decoder and a retrieval-based ranker. By leveraging a copy mechanism and a four-player adversarial game, IDGAN achieves state-of-the-art results on the MSCOCO dataset, producing captions that are both descriptive and syntactically natural.

Introduction

Automatically describing visual content—image captioning—is a cornerstone of computer vision and natural language processing. Currently, the field is divided into two paradigms:

  1. Generation-based methods: These utilize encoder-decoder architectures (e.g., CNN-LSTMs) to synthesize novel sentences. While flexible, they often produce generic or non-fluent "safe" captions.
  2. Retrieval-based methods: These search databases for existing captions. While grammatically perfect, they are constrained by the diversity of the training set and often fail to capture unique details in new images.

IDGAN addresses the lack of communication between these paradigms, aiming to "absorb the merits" of both by allowing the retrieval model to guide generation while the generation model provides synthetic candidates to refine the retrieval model's ranking capabilities.

Methodology: The IDGAN Architecture

The IDGAN framework consists of four interacting components designed to create a symmetric feedback loop:

The full IDGAN architecture, illustrating the interaction between the Generative Caption Decoder, Image Encoder, and the dual Ranking/Classification models.
  • Generative Seq2Seq Model ($G_{\theta1}$): A two-layer attention-based LSTM featuring a Copy Mechanism. This allows the model to "copy" informative words directly from retrieved "guidance captions," effectively handling out-of-vocabulary (OOV) words.
  • Generative Ranker ($G_{\theta2}$): Evaluates a pool of retrieved and synthetic captions to select adversarial candidates.
  • Discriminative Classifier ($D_{\phi1}$): A standard GAN discriminator that distinguishes human-written captions from machine-generated ones, incorporating CIDEr scores to reward adequacy.
  • Discriminative Ranker ($D_{\phi2}$): Learns to distinguish ground-truth image-caption pairs from adversarial pairs provided by both generators.

Dual Interaction and Reward Shaping

The two generators provide mutual feedback: the retrieval-based generator provides "guidance" to the generation model, while the generation model provides "synthetic candidates" to help the retrieval model learn better ranking features. Because the sampling process is non-differentiable, the system is trained using Policy Gradient (REINFORCE). The reward for the generator is a weighted combination of the binary classifier's output, the ranker's output, and CIDEr metrics.

Experimental Results

The model was evaluated on the MSCOCO 2014 dataset using the Karpathy split (113,287 training, 5,000 validation, and 5,000 testing images).

Quantitative Performance

IDGAN significantly outperformed existing state-of-the-art models across standard metrics:

  • CIDEr: Achieved 123.5, surpassing the TopDown baseline (120.1).
  • BLEU-4: Reached 38.5, a notable improvement over generation-only models.
  • Human Evaluation: In blind tests, IDGAN scored higher in both Informativeness (3.32) and Fluency (3.35) compared to strong baselines.

Qualitative Analysis

Qualitative results demonstrate that IDGAN produces more specific and contextually accurate descriptions than standard top-down attention models.

Qualitative comparison of captions generated by Ground Truth, TopDown, and IDGAN across various image contexts.

Discussion and Future Work

Ablation studies confirmed that both the copy mechanism and the discriminative ranker are essential for performance. However, the authors note two primary limitations:

  1. Semantic Diversity: The model can produce repetitive captions for visually similar images. Future work may involve developing rewards specifically for semantic diversity.
  2. Object Detection Dependency: The system's reliance on a pre-trained Faster R-CNN means that if the detector misses small objects, the captioner likely will too. Robust, end-to-end detection strategies tailored for captioning remain a promising area for future research.

Summary updated

Abstract

Image captioning is usually built on either generation-based or retrieval-based approaches. Both ways have certain strengths but suffer from their own limitations. In this paper, we propose an Interactive Dual Generative Adversarial Network (IDGAN) for image captioning, which mutually combines the retrieval-based and generation-based methods to learn a better image captioning ensemble. IDGAN consists of two generators and two discriminators, where the generation- and retrieval-based generators mutually benefit from each other's complementary targets that are learned from two dual adversarial discriminators. Specifically, the generation- and retrieval-based generators provide improved synthetic and retrieved candidate captions with informative feedback signals from the two respective discriminators that are trained to distinguish the generated captions from the true captions and assign top rankings to true captions respectively, thus featuring the merits of both retrieval-based and generation-based approaches. Extensive experiments on MSCOCO dataset demonstrate that the proposed IDGAN model significantly outperforms the compared methods for image captioning.
BibTeX
@inproceedings{liu2020interactive,
  author = {Junhao Liu and Kai Wang and Chunpu Xu and Zhou Zhao and Ruifeng Xu and Ying Shen and Min Yang},
  title = {{Interactive Dual Generative Adversarial Networks for Image Captioning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i07.6826},
}
Figure