Uncertainty-Aware Image Captioning

Zhengcong Fei, Mingyuan Fan, Li Zhu, Junshi Huang, Xiaoming Wei, Xiaolin Wei

Abstract

It is well believed that the higher uncertainty in a word of the caption, the more inter-correlated context information is required to determine it. However, current image captioning methods usually consider the generation of all words in a sentence sequentially and equally. In this paper, we propose an uncertainty-aware image captioning framework, which parallelly and iteratively operates insertion of discontinuous candidate words between existing words from easy to difficult until converged. We hypothesize that high-uncertainty words in a sentence need more prior information to make a correct decision and should be produced at a later stage. The resulting non-autoregressive hierarchy makes the caption generation explainable and intuitive. Specifically, we utilize an image-conditioned bag-of-word model to measure the word uncertainty and apply a dynamic programming algorithm to construct the training pairs. During inference, we devise an uncertainty-adaptive parallel beam search technique that yields an empirically logarithmic time complexity. Extensive experiments on the MS COCO benchmark reveal that our approach outperforms the strong baseline and related methods on both captioning quality as well as decoding speed.

Introduction

Image captioning, the task of generating natural language descriptions for visual content, has traditionally faced a trade-off between decoding speed and caption quality. Autoregressive Image Captioning (AIC) models generate words sequentially, ensuring high quality but suffering from linear time complexity ($O(N)$). Conversely, Non-Autoregressive Image Captioning (NAIC) models predict all words simultaneously for speed ($O(1)$), but often produce repetitive or incoherent results due to the lack of word-to-word conditioning.

The proposed Uncertainty-Aware Image Captioning (UAIC) framework bridges this gap by mimicking human intuition: identifying "easy" objects first and filling in "hard" relational details later. This approach recognizes that not all words are equally difficult to predict; concrete nouns like "people" often have lower uncertainty than prepositions like "with."

Comparison of generation strategies: Non-autoregressive (NAIC), Autoregressive (AIC), and the proposed Uncertainty-Aware (UAIC) approach.

Methodology

The UAIC framework introduces a novel "easy-to-hard" generation hierarchy through three primary components:

  • Uncertainty Estimator: An image-conditioned Bag-of-Words (BoW) model calculates the uncertainty of each word based on visual features alone. Words that are easily identifiable from the image receive low uncertainty scores and serve as the sentence's "anchor" words.
  • Insertion Transformer: Unlike standard decoders, this architecture allows for the parallel insertion of words into slots between existing words, enabling the sentence to grow non-monotonically.
  • DP-based Training: A Dynamic Programming algorithm partitions ground-truth captions into hierarchical stages based on word uncertainty. This ensures the model learns to prioritize informative "scaffold" words (like nouns) in early stages and functional words (like prepositions) in later stages.
The UAIC architecture, illustrating the interaction between the Uncertainty Estimator and the Insertion Transformer.

During inference, the model employs an Uncertainty-Adaptive Beam Search, which dynamically adjusts search breadth based on word complexity. This results in an empirical logarithmic time complexity ($O(\log N)$), as the sentence length can approximately double at each iteration.

Experimental Results

The model was evaluated on the industry-standard MS COCO benchmark using the Karpathy split. UAIC demonstrated superior performance across both quality and efficiency metrics:

  • Caption Quality: UAIC achieved a CIDEr score of 131.7, outperforming strong non-autoregressive baselines like SAIC (127.1) and rivaling top-tier autoregressive models.
  • Decoding Speed: The framework delivered a latency of 55ms, representing a 3.18x speedup over the standard autoregressive baseline.
  • Human Preference: In Turing-style evaluations, humans preferred UAIC captions over standard NAIC results by a significant margin (76.8% vs 60.5%), citing better logical coherence and fewer word repetitions.

Conclusion

UAIC introduces an explainable, hierarchical approach to image captioning that treats words according to their inherent difficulty. By prioritizing low-uncertainty words to provide context for more complex ones, the model achieves a balance of speed and accuracy that surpasses previous parallel methods. Future research may explore extending this uncertainty-aware insertion logic to other sequence generation tasks, such as machine translation or text summarization.

Summary updated

Abstract

It is well believed that the higher uncertainty in a word of the caption, the more inter-correlated context information is required to determine it. However, current image captioning methods usually consider the generation of all words in a sentence sequentially and equally. In this paper, we propose an uncertainty-aware image captioning framework, which parallelly and iteratively operates insertion of discontinuous candidate words between existing words from easy to difficult until converged. We hypothesize that high-uncertainty words in a sentence need more prior information to make a correct decision and should be produced at a later stage. The resulting non-autoregressive hierarchy makes the caption generation explainable and intuitive. Specifically, we utilize an image-conditioned bag-of-word model to measure the word uncertainty and apply a dynamic programming algorithm to construct the training pairs. During inference, we devise an uncertainty-adaptive parallel beam search technique that yields an empirically logarithmic time complexity. Extensive experiments on the MS COCO benchmark reveal that our approach outperforms the strong baseline and related methods on both captioning quality as well as decoding speed.
BibTeX
@inproceedings{fei2023uncertainty,
  author = {Zhengcong Fei and Mingyuan Fan and Li Zhu and Junshi Huang and Xiaoming Wei and Xiaolin Wei},
  title = {{Uncertainty-Aware Image Captioning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i1.25137},
}
Figure