Speaking the Same Language: Matching Machine to Human Captions by Adversarial Training

Rakshith Shetty, Marcus Rohrbach, Lisa Anne Hendricks, Mario Fritz, Bernt Schiele

Introduction

Current image captioning systems are remarkably proficient at identifying objects, yet they often lack the descriptive richness of human language. When presented with diverse images of a similar activity—such as skiing—standard models frequently default to a single, generic phrase like "a man riding skis down a snow covered slope." This "generic bias" stems from training objectives that maximize the likelihood of ground-truth data, leading models to favor the most frequent n-grams to minimize error.

This paper addresses three primary shortcomings in the field: exposure bias (where models rely on their own potentially flawed previous words during inference), generic bias (resulting in repetitive phrasing), and a lack of diversity (ignoring the multiple valid ways humans might describe a single scene).

Methods: The Adversarial Framework

The authors propose an Adversarial Caption Generator that utilizes a Generative Adversarial Network (GAN) to produce more human-like descriptions.

Architecture and Differentiability

The generator is built on a standard LSTM-based captioning model. However, generating text involves discrete word choices, which are typically non-differentiable and thus incompatible with standard backpropagation in GANs. To overcome this, the authors employ a Gumbel-Softmax approximation. This "soft-sampling" technique allows the model to approximate discrete choices in a continuous manner, making the entire pipeline end-to-end differentiable.

The adversarial training architecture featuring the Gumbel-Softmax sampler for end-to-end differentiability.

The Discriminator and Set-Based Evaluation

Unlike standard discriminators that evaluate single captions, this model's discriminator analyzes a set of captions for a single image. It uses distance kernels to assess two critical dimensions:

  1. Correctness: How well each caption aligns with the visual features of the image.
  2. Diversity: How much the captions within the set vary from one another.

By penalizing sets of identical, generic captions, the discriminator forces the generator to explore a wider vocabulary and more specific descriptive structures.

Key Results and Qualitative Impact

The adversarial model was evaluated on the MS-COCO dataset and demonstrated a significant shift toward linguistic richness compared to a maximum-likelihood baseline.

Quantitative Improvements

  • Vocabulary Expansion: The adversarial model utilized 2,176 unique words, nearly doubling the baseline's vocabulary of 1,085 words.
  • Novelty: Approximately 79.8% of the generated sentences were entirely new (not present in the training set), whereas the baseline only produced 44.2% novel sentences.
  • Human Preference: In head-to-head trials, human judges consistently preferred or rated adversarial captions as equal to the baseline, despite the adversarial model scoring lower on traditional automated metrics like METEOR. This suggests that standard metrics may actually penalize the very diversity that humans find valuable.

Qualitative Specificity

The model successfully moved beyond generic descriptions to capture specific actions and contexts. As illustrated below, where a baseline might provide a single repetitive caption for multiple images, the adversarial model identifies nuances such as "jumping over a ramp" or "cross country skiing."

Qualitative comparison showing how the adversarial model (Ours) provides specific, diverse captions for different skiing images compared to the repetitive baseline.

Discussion and Future Work

This research highlights a critical diversity-accuracy trade-off. While adversarial training might lead to a slight decrease in n-gram matching scores (like BLEU or CIDEr), it results in language that is statistically and qualitatively closer to human description.

Despite these gains, a significant gap remains: humans used over 7,000 unique words on the same test set, indicating that machine captioning still has a long way to go to match human creativity. Future work will likely focus on stabilizing GAN training for text and developing new automated metrics that better reflect human judgment of quality and diversity.

Summary updated

Abstract

While strong progress has been made in image captioning over the last years, machine and human captions are still quite distinct. A closer look reveals that this is due to the deficiencies in the generated word distribution, vocabulary size, and strong bias in the generators towards frequent captions. Furthermore, humans -- rightfully so -- generate multiple, diverse captions, due to the inherent ambiguity in the captioning task which is not considered in today's systems. To address these challenges, we change the training objective of the caption generator from reproducing groundtruth captions to generating a set of captions that is indistinguishable from human generated captions. Instead of handcrafting such a learning target, we employ adversarial training in combination with an approximate Gumbel sampler to implicitly match the generated distribution to the human one. While our method achieves comparable performance to the state-of-the-art in terms of the correctness of the captions, we generate a set of diverse captions, that are significantly less biased and match the word statistics better in several aspects.
BibTeX
@inproceedings{shetty2017speaking,
  author = {Rakshith Shetty and Marcus Rohrbach and Lisa Anne Hendricks and Mario Fritz and Bernt Schiele},
  title = {{Speaking the Same Language: Matching Machine to Human Captions by Adversarial Training}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2017},
}
Figure