Abstract
Image captioning has traditionally faced a trade-off between the high quality of slow Autoregressive (AR) models and the high speed of lower-quality Non-Autoregressive (NAR) models. This paper introduces Partially Non-Autoregressive Image Captioning (PNAIC), a hybrid framework that generates word groups in parallel while maintaining sequential dependencies within each group. Evaluated on the MS COCO dataset, PNAIC achieves a 3.5x speedup over AR baselines with virtually no loss in caption quality, effectively bridging the gap between sequential accuracy and parallel efficiency.
Introduction
Generating natural language descriptions for images—image captioning—is a core task in computer vision and NLP. Traditionally, this has relied on Autoregressive (AR) models, which generate text sequentially ($O(N)$ time complexity). While accurate, this sequential nature is a bottleneck for real-time industrial applications. Conversely, Non-Autoregressive (NAR) models attempt to predict all words simultaneously ($O(1)$ complexity). However, NAR models often suffer from the "multimodality problem," where independent word predictions lead to incoherence or repetitions (e.g., "a cat cat on the mat").
This paper proposes PNAIC as a "Goldilocks" solution: a model that is significantly faster than AR but more coherent than NAR by balancing local sequential logic with global parallel generation.
Methodology
The PNAIC model treats a caption as a series of concatenated word groups rather than a single sequence of independent tokens.
Group-Level Parallelism
The caption is divided into $K$ groups generated simultaneously. Within each group, words are generated one by one (locally autoregressive), resulting in a reduced time complexity of $O(N/K)$. This structure allows the model to maintain the linguistic "glue" required for local coherence while benefiting from parallel execution.
Key Architectural Components
- Two-Stage Position Encoding: To maintain global context, the model uses dual encodings: one for a word's position within its group and another for the group's index within the overall sentence.
- Curriculum Learning: The model utilizes an annealing strategy during training. It starts with random group splits to force the learning of dependencies and gradually moves toward uniform splits to optimize inference speed.
- Invalid Group Deletion: To mitigate the repetition common in parallel models, the authors introduced a special
DELtoken. If the model generates a redundant or nonsensical group, it learns to mark it for removal during post-processing.
Experimental Setup
The model was evaluated on the MS COCO dataset using the standard "Karpathy split" (113,287 training, 5,000 validation, and 5,000 test images). Performance was measured using standard NLP metrics, including BLEU-1/4, METEOR, ROUGE-L, CIDEr, and SPICE.
Results
PNAIC demonstrates a superior balance of speed and accuracy compared to existing paradigms:
- Inference Speed: With $K=5$, PNAIC achieved a latency of 49ms, a 3.59x speedup over the Autoregressive baseline (174ms).
- Caption Quality: On the MS COCO test set, PNAIC ($K=5$) achieved a CIDEr score of 128.5, matching the AR baseline and significantly outperforming pure NAR models like MNIC (108.1).
- Human Evaluation: In blind tests, PNAIC captions were rated as equal to or better than human annotations 76.3% of the time, slightly exceeding the AR baseline's 75.8%.
- Scalability: Even at $K=10$ (a 5.4x speedup), the model maintained a competitive CIDEr score of 125.2.
Conclusion
PNAIC provides a generalized framework for image captioning. By adjusting the hyperparameter $K$, the model can transition from a standard AR model ($K=1$) to a pure NAR model ($K=N$), allowing developers to tune the system based on specific hardware and latency requirements. Future work may explore dynamic group boundary determination and reinforcement learning to further enhance sentence-level consistency.
Summary updated