Abstract
Image captioning models often face a dilemma: curated datasets like MS COCO provide high accuracy but limited vocabulary, while web-scale datasets like Google Conceptual Captions (GCC) offer linguistic richness but suffer from significant noise. We propose Cooperative Distillation (co-distill), a framework that uses a BERT-based semantic bridge to dynamically weight training samples. By alternating between a denoising stream and a diversity stream, our model filters out incorrect captions while capturing expressive language. Human evaluations show our method achieves a Mean Opinion Score of 3.16, a substantial improvement over the 2.53 score of noisy baselines.
Introduction
In image captioning, the quality of training data is paramount. Curated datasets are "clean" but expensive and often result in repetitive, "safe" descriptions. Conversely, web-scale datasets are massive and linguistically diverse but contain "noisy" labels—captions that are grammatically incorrect or irrelevant to the image content. The goal of this work is to leverage the scale of noisy data without inheriting its flaws.
Methodology: Cooperative Distillation
The proposed framework consists of two Transformer-based models: a Student ($S$) trained on noisy data and a Teacher ($T$) trained on clean data. A Semantic Bridge, implemented using BERT embeddings, measures the similarity between ground-truth captions and model predictions to guide the learning process.
The Denoising Stream
In the denoising stream, the Student model evaluates noisy ground-truth captions against the Teacher's predictions. A weight ($w$) is calculated based on the semantic similarity (using BERT's [CLS] token) between the noisy label and the Teacher's output. If the similarity is high, the Student trusts the ground truth; if low, it relies more on the Teacher's "soft labels" through distillation.
The Diversity Stream
Uniquely, this framework features bidirectional learning. In the diversity stream, the Teacher learns from the Student. When the Student generates a caption that is semantically aligned with the clean ground truth but uses more diverse or expressive language, the Teacher incorporates this style. This prevents the Teacher from becoming overly terse or repetitive.
Results
The effectiveness of Cooperative Distillation was measured through human evaluation on Amazon MTurk, where participants rated captions on a 1-5 Likert scale.
Quantitative Performance
The Co-Distill method significantly outperformed the baseline model trained directly on noisy data. As shown in the comparison below, the framework effectively bridges the gap between clean and noisy data sources.
Qualitative Analysis
Qualitative results highlight the model's ability to ignore nonsensical web labels. For instance, where a noisy ground truth might provide an irrelevant phrase like "things to do before you move into a new house," the co-distilled model correctly identifies the scene as "a fire in a fireplace surrounded by logs."
Conclusion and Limitations
Cooperative Distillation provides a robust mechanism for denoising large-scale datasets while preserving linguistic variety. However, the study notes several areas for future work. The current implementation uses relatively small 2-layer Transformer architectures; scaling this approach to larger foundational models remains an open question. Additionally, the computational overhead of running BERT embeddings and alternating training streams is higher than standard methods.
Summary updated