Denoising Large-Scale Image Captioning from Alt-text Data using Content Selection Models

Khyathi Raghavi Chandu, Piyush Sharma, Soravit Changpinyo, Ashish Thapliyal, Radu Soricut

Abstract

Training large-scale image captioning (IC) models demands access to a rich and diverse set of training examples, gathered from the wild, often from noisy alt-text data. However, recent modeling approaches to IC often fall short in terms of performance in this case, because they assume a clean annotated dataset and employ an end-to-end generation approach, which often lacks both controllability and interpretability. We address these problems by breaking down the task into two simpler, more controllable tasks: skeleton prediction and skeleton-based caption generation. Specifically, we show that selecting content words as skeletons helps in generating improved and denoised captions when leveraging rich yet noisy alt-text-based uncurated datasets. We also show that the predicted English skeletons can be further cross-lingually leveraged to generate non-English captions, covering French, Italian, German, Spanish, and Hindi. Furthermore, skeleton-based prediction allows for better control of certain caption properties, such as length, content, and gender expression, providing a handle to perform human-in-the-loop semi-automatic corrections.

Introduction: The Challenge of Noisy Alt-Text

Standard image captioning models typically rely on massive datasets of images paired with human-written descriptions. To scale these models, researchers often harvest "alt-text" from the web. While abundant, alt-text is notoriously noisy; instead of describing the visual content of a photo, it often contains metadata or web-specific intent like "house for sale" or "click here for more."

End-to-end models trained on this data often "hallucinate" or repeat irrelevant web context because they cannot distinguish between a description of the image and the intent of the webpage. Moreover, these "black box" models offer little control over output style or inherent biases without expensive retraining.

The Skeleton Framework: A Two-Stage Approach

To address these issues, the researchers propose a two-stage framework that introduces an intermediate, interpretable layer: The Skeleton.

Comparison between baseline end-to-end captioning and the proposed skeleton-based approach, illustrating the denoising effect across multiple languages.
  1. Stage 1: Skeleton Prediction (Img2Ske): Instead of generating a full sentence immediately, the model predicts a sequence of lemmatized nouns and verbs (e.g., "collection," "book"). By restricting the output to key content words, the model naturally filters out the "meta-noise" common in alt-text.
  2. Stage 2: Skeleton-based Generation: A second model takes the image and the predicted skeleton to generate a fluent, descriptive caption. This stage ensures the final output is grammatically correct while remaining anchored to the visual objects identified in the skeleton.

Model Architecture

The authors implemented this framework using a Transformer-based architecture with three primary variants for integrating the skeleton information:

  • SkeEncoding: The skeleton is processed by a textual encoder and fed into the visual encoder as side information.
  • SkeDecoding: The skeleton is prepended to the caption within the decoder.
  • SkeAE (Autoencoding): A hybrid approach that simultaneously encodes and decodes the skeleton to reinforce content selection.
The detailed two-stage architecture featuring skeleton encoding (2a) and decoding (2b) modules within a Transformer framework.

Key Insights and Controllability

English as an Interlingua

A significant finding of the study is the utility of English skeletons as a cross-lingual bridge. By predicting an English skeleton and using it to generate captions in other languages (like Hindi or Italian), the model leverages the massive volume of English training data to improve performance in lower-resource languages.

Human-in-the-Loop Correction

Because the skeleton is a simple list of words, it provides a "handle" for intervention. The researchers demonstrated that by automatically replacing gendered terms in the skeleton (e.g., changing "man" to "person"), they could reduce gendered output by 46% in English and 70% in Hindi without any retraining of the underlying models.

Results and Evaluation

In side-by-side human evaluations on the T2 test set, the SkeEncoding model outperformed the standard baseline by a substantial margin of +11.0 gain. The approach significantly shifted caption distributions toward "Visible" relations (describing what is actually in the image) and away from "Meta" or "Story" relations (contextual fluff).

While the model showed gains in 4 out of 5 tested languages, a slight performance dip in Spanish highlighted the sensitivity of these models to regional dialect mismatches between training data and human evaluators.

Conclusion

By breaking image captioning into content selection and surface realization, this research provides a robust method for denoising web-scale data. The skeleton-based approach not only improves caption quality but also introduces much-needed interpretability and a mechanism for mitigating model bias in a multi-lingual context.

Summary updated

Abstract

Training large-scale image captioning (IC) models demands access to a rich and diverse set of training examples, gathered from the wild, often from noisy alt-text data. However, recent modeling approaches to IC often fall short in terms of performance in this case, because they assume a clean annotated dataset (as opposed to the noisier alt-text--based annotations), and employ an end-to-end generation approach, which often lacks both controllability and interpretability. We address these problems by breaking down the task into two simpler, more controllable tasks -- skeleton prediction and skeleton-based caption generation. Specifically, we show that selecting content words as skeletons} helps in generating improved and denoised captions when leveraging rich yet noisy alt-text--based uncurated datasets. We also show that the predicted English skeletons can be further cross-lingually leveraged to generate non-English captions, and present experimental results covering caption generation in French, Italian, German, Spanish and Hindi. We also show that skeleton-based prediction allows for better control of certain caption properties, such as length, content, and gender expression, providing a handle to perform human-in-the-loop semi-automatic corrections.
BibTeX
@misc{chandu2020denoising,
  author = {Khyathi Raghavi Chandu and Piyush Sharma and Soravit Changpinyo and Ashish Thapliyal and Radu Soricut},
  title = {{Denoising Large-Scale Image Captioning from Alt-text Data using Content Selection Models}},
  year = {2020},
  eprint = {2009.05175},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2009.05175},
}
Figure