Introduction
Large-scale web-crawled datasets are essential for training modern image captioning models. However, these datasets are inherently "noisy," containing many image-text pairs that are only partially matched or entirely unrelated. For instance, a caption might describe a "family of elephants" for an image that actually depicts a "dirt road."
Traditional denoising methods typically operate at a coarse-grained level, assigning a single weight to an entire sentence. This approach is inefficient because even a "noisy" caption often contains several accurate, useful words. The Noise-aware Image Captioning (NIC) framework addresses this by performing fine-grained denoising: identifying specific mismatched words and progressively replacing them with reliable model-generated "pseudo-labels" as training advances.
The NIC Framework
The NIC framework is a model-agnostic wrapper that can be applied to various state-of-the-art architectures. It employs a three-pronged strategy to identify and mitigate word-level noise.
1. Inter-Modal Representativeness
The model evaluates its own uncertainty regarding a word given the visual features of the image. By measuring the entropy of word predictions, the framework can flag words with high uncertainty as potential mismatches.
2. Intra-Modal Discriminativeness
This "look-ahead" strategy utilizes the decoder's sensitivity to context. The model feeds a word back into the decoder to observe its impact on subsequent word predictions. If a word is contextually incorrect, it triggers an error accumulation that makes the remainder of the sentence significantly harder to predict. This performance drop serves as a diagnostic signal for noise.
3. Progressive Coordination
Early in the training process, the model lacks the visual-linguistic grounding to distinguish right from wrong, so it initially trusts the original labels. As the model converges, a scheduling function gradually shifts trust away from suspicious original words and toward the model's own "pseudo-labels." This curriculum-based approach prevents the model from over-fitting to errors while avoiding early-stage hallucinations.
Experimental Evaluation
The researchers evaluated NIC on two primary benchmarks:
- MS-COCO: Used for controlled experiments with synthetic noise (uniform and asymmetric) at ratios from 20% to 80%.
- Conceptual Captions (CC152K): A subset of a real-world web-crawled dataset containing natural noise (estimated 3%–20%).
Key Results
- Robustness to High Noise: On MS-COCO with an 80% noise ratio, NIC improved the CIDEr score by 6.2 points over the PureT baseline.
- Real-World Performance: On the naturally noisy CC152K dataset, NIC outperformed existing state-of-the-art models like SCD-Net and DLCT across all standard metrics (BLEU, METEOR, ROUGE-L, CIDEr, and SPICE).
- Versatility: The framework consistently boosted performance when applied to different base models, including AoANet, RSTNet, and DLCT.
Conclusion and Future Work
The core innovation of NIC is the shift from sentence-level to word-level denoising. By treating error accumulation as a feature rather than a bug, the framework effectively salvages useful information from imperfect data.
While the "Subsequent Prediction Inspection" adds computational overhead, the gains in data efficiency and robustness are significant. Future research may extend this fine-grained analysis to other multi-modal domains, such as video captioning or visual question answering, where web-scale noise remains a primary challenge.
Summary updated