Noise-Aware Image Captioning with Progressively Exploring Mismatched Words

Zhongtian Fu, Kefei Song, Luping Zhou, Yang Yang

Introduction

Large-scale web-crawled datasets are essential for training modern image captioning models. However, these datasets are inherently "noisy," containing many image-text pairs that are only partially matched or entirely unrelated. For instance, a caption might describe a "family of elephants" for an image that actually depicts a "dirt road."

Examples of fine-grained noise in web-crawled image-caption pairs. Red indicates mismatched words, while blue indicates correctly matched words.

Traditional denoising methods typically operate at a coarse-grained level, assigning a single weight to an entire sentence. This approach is inefficient because even a "noisy" caption often contains several accurate, useful words. The Noise-aware Image Captioning (NIC) framework addresses this by performing fine-grained denoising: identifying specific mismatched words and progressively replacing them with reliable model-generated "pseudo-labels" as training advances.

The NIC Framework

The NIC framework is a model-agnostic wrapper that can be applied to various state-of-the-art architectures. It employs a three-pronged strategy to identify and mitigate word-level noise.

The architecture of the NIC framework, featuring Inter-Modal representativeness and Intra-Modal discriminativeness paths for noise detection.

1. Inter-Modal Representativeness

The model evaluates its own uncertainty regarding a word given the visual features of the image. By measuring the entropy of word predictions, the framework can flag words with high uncertainty as potential mismatches.

2. Intra-Modal Discriminativeness

This "look-ahead" strategy utilizes the decoder's sensitivity to context. The model feeds a word back into the decoder to observe its impact on subsequent word predictions. If a word is contextually incorrect, it triggers an error accumulation that makes the remainder of the sentence significantly harder to predict. This performance drop serves as a diagnostic signal for noise.

3. Progressive Coordination

Early in the training process, the model lacks the visual-linguistic grounding to distinguish right from wrong, so it initially trusts the original labels. As the model converges, a scheduling function gradually shifts trust away from suspicious original words and toward the model's own "pseudo-labels." This curriculum-based approach prevents the model from over-fitting to errors while avoiding early-stage hallucinations.

Experimental Evaluation

The researchers evaluated NIC on two primary benchmarks:

  • MS-COCO: Used for controlled experiments with synthetic noise (uniform and asymmetric) at ratios from 20% to 80%.
  • Conceptual Captions (CC152K): A subset of a real-world web-crawled dataset containing natural noise (estimated 3%–20%).

Key Results

  • Robustness to High Noise: On MS-COCO with an 80% noise ratio, NIC improved the CIDEr score by 6.2 points over the PureT baseline.
  • Real-World Performance: On the naturally noisy CC152K dataset, NIC outperformed existing state-of-the-art models like SCD-Net and DLCT across all standard metrics (BLEU, METEOR, ROUGE-L, CIDEr, and SPICE).
  • Versatility: The framework consistently boosted performance when applied to different base models, including AoANet, RSTNet, and DLCT.

Conclusion and Future Work

The core innovation of NIC is the shift from sentence-level to word-level denoising. By treating error accumulation as a feature rather than a bug, the framework effectively salvages useful information from imperfect data.

While the "Subsequent Prediction Inspection" adds computational overhead, the gains in data efficiency and robustness are significant. Future research may extend this fine-grained analysis to other multi-modal domains, such as video captioning or visual question answering, where web-scale noise remains a primary challenge.

Summary updated

Abstract

Image captioning aims to automatically generate captions for images by learning a cross-modal generator from vision to language. The large amount of image-text pairs required for training is usually sourced from the internet due to the manual cost, which brings the noise with mismatched relevance that affects the learning process. Unlike traditional noisy label learning, the key challenge in processing noisy image-text pairs is to finely identify the mismatched words to make the most use of trustworthy information in the text, rather than coarsely weighing the entire examples. To tackle this challenge, we propose a Noise-aware Image Captioning method (NIC) to adaptively mitigate the erroneous guidance from noise by progressively exploring mismatched words. Specifically, NIC first identifies mismatched words by quantifying word-label reliability from two aspects: 1) inter-modal representativeness, which measures the significance of the current word by assessing cross-modal correlation via prediction certainty; 2) intra-modal informativeness, which amplifies the effect of current prediction by combining the quality of subsequent word generation. During optimization, NIC constructs the pseudo-word-labels considering the reliability of the origin word-labels and model convergence to periodically coordinate mismatched words. As a result, NIC can effectively exploit both clean and noisy image-text pairs to learn a more robust mapping function. Extensive experiments conducted on the MS-COCO and Conceptual Caption datasets validate the effectiveness of our method in various noisy scenarios.
BibTeX
@inproceedings{fu2024noise,
  author = {Zhongtian Fu and Kefei Song and Luping Zhou and Yang Yang},
  title = {{Noise-Aware Image Captioning with Progressively Exploring Mismatched Words}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i11.29097},
}
Figure