Abstract
Traditional image captioning models face a trade-off between the high accuracy of slow autoregressive generation and the speed of less coherent non-autoregressive methods. The Uncertainty-Aware Image Captioning (UAIC) framework resolves this by generating captions in stages based on word "uncertainty." By prioritizing high-confidence objects and filling in complex relational words later, UAIC achieves a ~3x speedup over standard models while maintaining state-of-the-art quality.
Introduction
Image captioning has long been dominated by two paradigms:
- Autoregressive (AIC) models: These generate text word-by-word from left to right. While accurate due to strong sequential dependencies, they suffer from $O(N)$ time complexity.
- Non-autoregressive (NAIC) models: These attempt to generate all words simultaneously for $O(1)$ speed but often struggle with coherence and repetition.
UAIC introduces a middle ground based on the observation that not all words are equally difficult to predict. Words representing concrete objects (e.g., "people," "field") have low uncertainty, whereas relational words (e.g., "in," "with") require context. UAIC generates captions through a "hierarchy of difficulty," as illustrated below.
Methodology
The UAIC framework enhances the standard Transformer architecture with four key components:
- Uncertainty Estimator: Uses an image-conditioned Bag-of-Words (BoW) model to assign uncertainty scores to potential words.
- DP-based Training: A Dynamic Programming algorithm decomposes ground-truth captions into insertion steps, training the model to prioritize low-uncertainty words.
- Insertion Transformer: A decoder that predicts new words to "slot" into gaps between existing ones, rather than appending them to the end.
- Uncertainty-Adaptive Beam Search: Dynamically adjusts search width during inference—using faster, narrow searches for easy words and wider searches for complex relations.
Experimental Setup
The model was evaluated on the MS COCO benchmark (Karpathy split), comprising over 113,000 training images. Performance was compared against leading autoregressive (e.g., AoANet) and non-autoregressive (e.g., MNIC) baselines.
Key Insights
- Contextual Prerequisites: High-uncertainty words depend on the presence of low-uncertainty anchors. For instance, "on" is only predictable once "cat" and "mat" are identified.
- Logarithmic Efficiency: By generating multiple words in parallel at each stage, UAIC achieves an empirical time complexity of $O(\log N)$, a significant improvement over linear-time models.
- Explainability: The multi-stage process provides a window into the model’s reasoning, showing it first identifying actors and then refining their attributes and relationships.
Key Results
UAIC demonstrates that speed does not have to come at the cost of quality:
- Quality: Achieved a CIDEr score of 131.7, outperforming many autoregressive models and setting a new bar for non-autoregressive approaches.
- Efficiency: Delivered a 3.18x speedup over its autoregressive counterpart, with a latency of just 55ms per image.
- Human Evaluation: In Turing-style tests, annotators found UAIC captions nearly as natural as those from slower AIC models and far superior to standard NAIC outputs.
Conclusion
UAIC breaks the traditional left-to-right generation mold by adopting a more "natural" uncertainty-based ordering. While currently relying on a learned hierarchy, future work could explore even more flexible, non-monotonic generation strategies across various multi-modal tasks.
Summary updated