Alleviating Noisy Data in Image Captioning with Cooperative Distillation

Pierre L. Dognin, Igor Melnyk, Youssef Mroueh, Inkit Padhi, Mattia Rigotti, Jarret Ross, Yair SchiffarXiv 2020 · arXiv

Abstract

Image captioning models often face a dilemma: curated datasets like MS COCO provide high accuracy but limited vocabulary, while web-scale datasets like Google Conceptual Captions (GCC) offer linguistic richness but suffer from significant noise. We propose Cooperative Distillation (co-distill), a framework that uses a BERT-based semantic bridge to dynamically weight training samples. By alternating between a denoising stream and a diversity stream, our model filters out incorrect captions while capturing expressive language. Human evaluations show our method achieves a Mean Opinion Score of 3.16, a substantial improvement over the 2.53 score of noisy baselines.

Introduction

In image captioning, the quality of training data is paramount. Curated datasets are "clean" but expensive and often result in repetitive, "safe" descriptions. Conversely, web-scale datasets are massive and linguistically diverse but contain "noisy" labels—captions that are grammatically incorrect or irrelevant to the image content. The goal of this work is to leverage the scale of noisy data without inheriting its flaws.

Methodology: Cooperative Distillation

The proposed framework consists of two Transformer-based models: a Student ($S$) trained on noisy data and a Teacher ($T$) trained on clean data. A Semantic Bridge, implemented using BERT embeddings, measures the similarity between ground-truth captions and model predictions to guide the learning process.

The Denoising Stream

In the denoising stream, the Student model evaluates noisy ground-truth captions against the Teacher's predictions. A weight ($w$) is calculated based on the semantic similarity (using BERT's [CLS] token) between the noisy label and the Teacher's output. If the similarity is high, the Student trusts the ground truth; if low, it relies more on the Teacher's "soft labels" through distillation.

The Denoising Stream architecture showing the integration of Cross-Entropy (CE) and Kullback-Leibler (KL) divergence weighted by semantic similarity.

The Diversity Stream

Uniquely, this framework features bidirectional learning. In the diversity stream, the Teacher learns from the Student. When the Student generates a caption that is semantically aligned with the clean ground truth but uses more diverse or expressive language, the Teacher incorporates this style. This prevents the Teacher from becoming overly terse or repetitive.

Results

The effectiveness of Cooperative Distillation was measured through human evaluation on Amazon MTurk, where participants rated captions on a 1-5 Likert scale.

Quantitative Performance

The Co-Distill method significantly outperformed the baseline model trained directly on noisy data. As shown in the comparison below, the framework effectively bridges the gap between clean and noisy data sources.

Mean Opinion Score (MOS) comparison between the Co-Distill framework and the noisy baseline.

Qualitative Analysis

Qualitative results highlight the model's ability to ignore nonsensical web labels. For instance, where a noisy ground truth might provide an irrelevant phrase like "things to do before you move into a new house," the co-distilled model correctly identifies the scene as "a fire in a fireplace surrounded by logs."

Conclusion and Limitations

Cooperative Distillation provides a robust mechanism for denoising large-scale datasets while preserving linguistic variety. However, the study notes several areas for future work. The current implementation uses relatively small 2-layer Transformer architectures; scaling this approach to larger foundational models remains an open question. Additionally, the computational overhead of running BERT embeddings and alternating training streams is higher than standard methods.

Summary updated

Abstract

Image captioning systems have made substantial progress, largely due to the availability of curated datasets like Microsoft COCO or Vizwiz that have accurate descriptions of their corresponding images. Unfortunately, scarce availability of such cleanly labeled data results in trained algorithms producing captions that can be terse and idiosyncratically specific to details in the image. We propose a new technique, cooperative distillation that combines clean curated datasets with the web-scale automatically extracted captions of the Google Conceptual Captions dataset (GCC), which can have poor descriptions of images, but is abundant in size and therefore provides a rich vocabulary resulting in more expressive captions.
BibTeX
@misc{dognin2020alleviating,
  author = {Pierre L. Dognin and Igor Melnyk and Youssef Mroueh and Inkit Padhi and Mattia Rigotti and Jarret Ross and Yair Schiff},
  title = {{Alleviating Noisy Data in Image Captioning with Cooperative Distillation}},
  year = {2020},
  eprint = {2012.11691},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2012.11691},
}
Figure