Abstract
In this paper we study image captioning as a conditional GAN training, proposing both a context-aware LSTM captioner and co-attentive discriminator, which enforces semantic alignment between images and captions. We empirically focus on the viability of two training methods: Self-critical Sequence Training (SCST) and Gumbel Straight-Through (ST) and demonstrate that SCST shows more stable gradient behavior and improved results over Gumbel ST, even without accessing discriminator gradients directly. We also address the problem of automatic evaluation for captioning models and introduce a new semantic score, and show its correlation to human judgement. As an evaluation paradigm, we argue that an important criterion for a captioner is the ability to generalize to compositions of objects that do not usually co-occur together. To this end, we introduce a small captioned Out of Context (OOC) test set. The OOC set, combined with our semantic score, are the proposed new diagnosis tools for the captioning community. When evaluated on OOC and MS-COCO benchmarks, we show that SCST-based training has a strong performance in both semantic score and human evaluation, promising to be a valuable new approach for efficient discrete GAN training.
Introduction
Most image captioning models are traditionally trained to maximize n-gram overlap metrics like CIDEr or BLEU. While computationally efficient, these metrics fail to capture the semantic coherence between an image and its description, often leading to "safe" but generic captions. Furthermore, these models frequently suffer from dataset bias, struggling to describe unusual or "out-of-context" scenes because they rely on memorized word patterns rather than true visual understanding.
This paper introduces a novel framework for image captioning using Generative Adversarial Networks (GANs) to move beyond simple word-matching. By combining a context-aware generator with a co-attentive discriminator and a stable training method, the authors produce captions that are more diverse, natural, and semantically aligned with images.
Methods
The proposed architecture consists of two primary components designed to enhance semantic alignment:
- Context-Aware Captioner (Generator): This LSTM-based model utilizes an "adaptive visual sentinel." Unlike standard models, it feeds a mixture of image features and past attention contexts back into the LSTM at every step, ensuring the model maintains a persistent memory of the visual regions it has already processed.
- Co-attention Pooling Discriminator: Rather than comparing a global sentence embedding to a global image embedding, this discriminator employs a co-attention mechanism. It explicitly models the relationship between specific words and specific image regions to distinguish between human-written "real" captions and machine-generated "fake" ones.
To address the instability of discrete GAN training, the authors compared Self-critical Sequence Training (SCST)—a variant of reinforcement learning—against the Gumbel Straight-Through method. SCST was found to provide significantly more stable gradients, preventing the model collapse often associated with adversarial text generation.
Datasets and Evaluation
The models were evaluated on the industry-standard MS-COCO benchmark and the SBU Captions dataset. Crucially, the authors introduced the Out of Context (OOC) set: a diagnostic dataset containing 269 images of objects in unusual settings (e.g., a bed in a forest). This set tests whether a model has actually learned to ground language in vision or is merely memorizing common co-occurrences.
To complement this, a new Semantic Score was developed using Canonical Correlation Analysis (CCA) to measure the similarity between image and text embeddings. This metric correlates more closely with human judgment than traditional n-gram metrics.
Key Results
The GAN-based approach demonstrated superior generalization and linguistic diversity compared to standard baselines:
- Generalization to OOC: On the diagnostic OOC dataset, the attentive GAN models significantly outperformed non-attentional baselines. As shown in the training curves, the proposed models (GAN1 and GAN2) achieved higher semantic scores, proving that the co-attention mechanism helps the model describe novel scenes.
- Linguistic Diversity: While standard Reinforcement Learning (CIDEr-RL) often results in "vocabulary collapse" (dropping to 5.1% coverage), the SCST-GAN models maintained a coverage of approximately 11%, leading to more descriptive and human-like language.
- Human Evaluation: In human preference tests, the GAN ensemble was frequently preferred over standard Cross-Entropy and RL baselines, with evaluators often finding the GAN-generated captions indistinguishable from human ones.
Conclusion
While the GAN approach sometimes results in lower CIDEr scores than pure RL models, it produces captions that are more natural and semantically grounded. The introduction of the OOC dataset and the Semantic Score provides the community with essential tools for diagnosing model bias. Future work will focus on scaling these diagnostic tools and further refining the balance between reference-string matching and natural description.
Summary updated