Uncertainty-Aware Image Captioning

Zhengcong Fei, Mingyuan Fan, Li Zhu, Junshi Huang, Xiaoming Wei, Xiaolin Wei

Abstract

Traditional image captioning models face a trade-off between the high accuracy of slow autoregressive generation and the speed of less coherent non-autoregressive methods. The Uncertainty-Aware Image Captioning (UAIC) framework resolves this by generating captions in stages based on word "uncertainty." By prioritizing high-confidence objects and filling in complex relational words later, UAIC achieves a ~3x speedup over standard models while maintaining state-of-the-art quality.

Introduction

Image captioning has long been dominated by two paradigms:

  1. Autoregressive (AIC) models: These generate text word-by-word from left to right. While accurate due to strong sequential dependencies, they suffer from $O(N)$ time complexity.
  2. Non-autoregressive (NAIC) models: These attempt to generate all words simultaneously for $O(1)$ speed but often struggle with coherence and repetition.

UAIC introduces a middle ground based on the observation that not all words are equally difficult to predict. Words representing concrete objects (e.g., "people," "field") have low uncertainty, whereas relational words (e.g., "in," "with") require context. UAIC generates captions through a "hierarchy of difficulty," as illustrated below.

Comparison of captioning paradigms: NAIC (one step), AIC (sequential), and UAIC (uncertainty-based stages).

Methodology

The UAIC framework enhances the standard Transformer architecture with four key components:

  • Uncertainty Estimator: Uses an image-conditioned Bag-of-Words (BoW) model to assign uncertainty scores to potential words.
  • DP-based Training: A Dynamic Programming algorithm decomposes ground-truth captions into insertion steps, training the model to prioritize low-uncertainty words.
  • Insertion Transformer: A decoder that predicts new words to "slot" into gaps between existing ones, rather than appending them to the end.
  • Uncertainty-Adaptive Beam Search: Dynamically adjusts search width during inference—using faster, narrow searches for easy words and wider searches for complex relations.

Experimental Setup

The model was evaluated on the MS COCO benchmark (Karpathy split), comprising over 113,000 training images. Performance was compared against leading autoregressive (e.g., AoANet) and non-autoregressive (e.g., MNIC) baselines.

Key Insights

  • Contextual Prerequisites: High-uncertainty words depend on the presence of low-uncertainty anchors. For instance, "on" is only predictable once "cat" and "mat" are identified.
  • Logarithmic Efficiency: By generating multiple words in parallel at each stage, UAIC achieves an empirical time complexity of $O(\log N)$, a significant improvement over linear-time models.
  • Explainability: The multi-stage process provides a window into the model’s reasoning, showing it first identifying actors and then refining their attributes and relationships.

Key Results

UAIC demonstrates that speed does not have to come at the cost of quality:

  • Quality: Achieved a CIDEr score of 131.7, outperforming many autoregressive models and setting a new bar for non-autoregressive approaches.
  • Efficiency: Delivered a 3.18x speedup over its autoregressive counterpart, with a latency of just 55ms per image.
  • Human Evaluation: In Turing-style tests, annotators found UAIC captions nearly as natural as those from slower AIC models and far superior to standard NAIC outputs.

Conclusion

UAIC breaks the traditional left-to-right generation mold by adopting a more "natural" uncertainty-based ordering. While currently relying on a learned hierarchy, future work could explore even more flexible, non-monotonic generation strategies across various multi-modal tasks.

Summary updated

Abstract

It is well believed that the higher uncertainty in a word of the caption, the more inter-correlated context information is required to determine it. However, current image captioning methods usually consider the generation of all words in a sentence sequentially and equally. In this paper, we propose an uncertainty-aware image captioning framework, which parallelly and iteratively operates insertion of discontinuous candidate words between existing words from easy to difficult until converged. We hypothesize that high-uncertainty words in a sentence need more prior information to make a correct decision and should be produced at a later stage. The resulting non-autoregressive hierarchy makes the caption generation explainable and intuitive. Specifically, we utilize an image-conditioned bag-of-word model to measure the word uncertainty and apply a dynamic programming algorithm to construct the training pairs. During inference, we devise an uncertainty-adaptive parallel beam search technique that yields an empirically logarithmic time complexity. Extensive experiments on the MS COCO benchmark reveal that our approach outperforms the strong baseline and related methods on both captioning quality as well as decoding speed.
BibTeX
@inproceedings{fei2023uncertainty,
  author = {Zhengcong Fei and Mingyuan Fan and Li Zhu and Junshi Huang and Xiaoming Wei and Xiaolin Wei},
  title = {{Uncertainty-Aware Image Captioning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i1.25137},
}
Figure