Speaking the Same Language: Matching Machine to Human Captions by Adversarial Training

Rakshith Shetty, Marcus Rohrbach, Lisa Anne Hendricks, Mario Fritz, Bernt Schiele

Abstract

While strong progress has been made in image captioning over the last years, machine and human captions are still quite distinct. A closer look reveals that this is due to the deficiencies in the generated word distribution, vocabulary size, and strong bias in the generators towards frequent captions. Furthermore, humans -- rightfully so -- generate multiple, diverse captions, due to the inherent ambiguity in the captioning task which is not considered in today's systems. To address these challenges, we change the training objective of the caption generator from reproducing groundtruth captions to generating a set of captions that is indistinguishable from human generated captions. Instead of handcrafting such a learning target, we employ adversarial training in combination with an approximate Gumbel sampler to implicitly match the generated distribution to the human one. While our method achieves comparable performance to the state-of-the-art in terms of the correctness of the captions, we generate a set of diverse captions, that are significantly less biased and match the word statistics better in several aspects.

Introduction

Modern image captioning systems have reached high levels of descriptive accuracy, yet they often lack the natural variety and vocabulary richness of human speech. Most state-of-the-art models are trained using Maximum Likelihood Estimation (MLE), which encourages the model to predict the most probable next word based on training data. This "safe" strategy leads to a significant bias toward frequent words and phrases, resulting in generic and repetitive captions. For example, a model might describe almost any skiing scene with the same phrase: "a man riding skis down a snow covered slope."

Furthermore, standard captioning models fail to account for the inherent ambiguity of visual scenes. A single image can be described in numerous valid ways depending on the observer's focus. Humans naturally produce diverse captions, whereas MLE-based models struggle to generate multiple distinct options for the same image. This paper proposes a framework to bridge this gap by matching the statistical distribution of machine-generated captions to those of humans.

Methodology: Adversarial Training for Text

To address the limitations of MLE, the authors frame image captioning as an adversarial game using a Generative Adversarial Network (GAN). The goal is to train a generator that produces captions indistinguishable from human ones to a discriminator.

Overcoming the Differentiability Barrier

A primary challenge in applying GANs to text is the discrete nature of word tokens. Standard backpropagation requires continuous functions, but the sampling of words is a non-differentiable operation. The authors overcome this using the Gumbel-Softmax approximation. This technique provides a continuous relaxation of the discrete sampling process, allowing gradients to flow from the discriminator back to the generator during training.

The Set-Aware Discriminator

To prevent "mode collapse"—where the generator learns a single high-quality caption and repeats it for every image—the authors introduce a set-aware discriminator. Instead of evaluating a single caption, the discriminator analyzes a set of multiple captions for a given image. By using distance-measuring kernels, the discriminator evaluates both the accuracy of the descriptions relative to the image and the diversity within the set. This forces the generator to explore a wider range of the vocabulary and produce varied sentences.

Training Stability

Adversarial training for discrete sequences is notoriously unstable. To mitigate this, the authors incorporate a feature matching loss. This objective requires the generator to match the expected intermediate activations of the discriminator for both real (human) and generated data, providing a more stable learning signal than the raw discriminator output alone.

Experimental Results

The model was evaluated on the MS-COCO dataset, a standard benchmark for image captioning. While the adversarial model scored lower on traditional n-gram matching metrics like METEOR and SPICE, these metrics were found to be poor indicators of quality for diverse models. N-gram metrics reward the repetition of common phrases, which the adversarial model explicitly avoids.

Vocabulary and Novelty

The adversarial approach significantly expanded the model's vocabulary. Compared to the baseline's 1,085 unique words, the adversarial model utilized up to 2,671 unique words. Furthermore, the model generated entirely novel sentences—those not present in the training set—72.5% of the time, nearly doubling the baseline's novelty rate of 44.2%.

Comparison of captions generated by the proposed adversarial model versus a baseline MLE model.

Human Evaluation

The true test of the model's performance was conducted via human judgment on Amazon Mechanical Turk. In blind A/B tests, humans preferred the adversarial captions 36.9% of the time, compared to 34.8% for the baseline. This confirms that the increase in diversity and vocabulary did not come at the expense of descriptive accuracy.

Conclusion and Future Work

This research demonstrates that adversarial training, combined with set-aware discrimination and Gumbel-Softmax sampling, can produce image captions that are more human-like in their diversity and statistical distribution. While a gap still remains between machine and human vocabulary sizes (2,671 vs 7,253 unique words in the test set), this work provides a robust framework for moving beyond the repetitive nature of MLE-based systems. Future research may focus on further stabilizing GAN training for text and exploring methods to capture the full breadth of human linguistic expression.

Summary updated

Abstract

While strong progress has been made in image captioning over the last years, machine and human captions are still quite distinct. A closer look reveals that this is due to the deficiencies in the generated word distribution, vocabulary size, and strong bias in the generators towards frequent captions. Furthermore, humans -- rightfully so -- generate multiple, diverse captions, due to the inherent ambiguity in the captioning task which is not considered in today's systems. To address these challenges, we change the training objective of the caption generator from reproducing groundtruth captions to generating a set of captions that is indistinguishable from human generated captions. Instead of handcrafting such a learning target, we employ adversarial training in combination with an approximate Gumbel sampler to implicitly match the generated distribution to the human one. While our method achieves comparable performance to the state-of-the-art in terms of the correctness of the captions, we generate a set of diverse captions, that are significantly less biased and match the word statistics better in several aspects.
BibTeX
@inproceedings{shetty2017speaking,
  author = {Rakshith Shetty and Marcus Rohrbach and Lisa Anne Hendricks and Mario Fritz and Bernt Schiele},
  title = {{Speaking the Same Language: Matching Machine to Human Captions by Adversarial Training}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2017},
}
Figure