Image Captioning via Compact Bidirectional Architecture

Yuanen Zhou, Zhenzhen Hu, Daqing Liu, Huixia Ben, Meng Wang, Hong, Richang, Wang, Meng

TL;DR

This paper introduces a novel "Compact Bidirectional Transformer" (CBTrans) architecture for image captioning that allows models to leverage both past and future context within a single, efficient decoder. This approach achieves state-of-the-art results by implicitly and explicitly incorporating bidirectional information, outperforming unidirectional models and existing refinement-based methods.

What Problem Does This Paper Address?

Traditional image captioning models generate sentences word-by-word from left to right. This unidirectional approach limits their ability to consider future context, potentially leading to less accurate or descriptive captions. While some "refinement-based" models attempt to address this by using a second stage to revise an initial caption, they typically employ two separate networks that must run sequentially, hindering parallel processing and efficiency. The authors aim to bridge this gap by developing a single, compact model that can effectively utilize bidirectional context for improved caption generation while maintaining parallelizable decoding.

Methods

The core innovation is the Compact Bidirectional Transformer (CBTrans). Instead of separate networks, CBTrans tightly couples left-to-right (L2R) and right-to-left (R2L) decoding flows within a single Transformer architecture. This is achieved by:

  • Shared Parameters: The L2R and R2L decoders share parameters, making the model more efficient.
  • Implicit Bidirectional Context: The architecture inherently encourages the model to learn from both directions, acting as a form of regularization.
  • Optional Explicit Interaction: A "masked multi-head bidirectional interactive attention" sublayer allows for explicit interaction between the L2R and R2L flows, though ablation studies suggest this is less critical than the overall architecture.
  • Parallel Decoding: The unified decoder allows for parallel execution of both L2R and R2L flows, leveraging GPU capabilities.
  • Sentence-Level Ensemble: During inference, captions generated by the L2R and R2L flows are ranked, and the better one is selected. This is a key component that contributes significantly to performance.
  • Two-Flow Self-Critical Training: The authors extend the standard self-critical training (SC) method to accommodate the two bidirectional flows.
  • CBLSTM Model: The authors demonstrate the generality of their approach by extending it to an LSTM backbone, creating the CBLSTM model.

Datasets

The primary dataset used for training and evaluation is MSCOCO. The authors utilize the "Karpathy splits" for offline experiments, which include training, validation, and testing sets. They also report results on the MSCOCO official testing set for online evaluation.

Key Insights and Ideas

  • Bidirectional Context is Crucial: The paper strongly advocates for the importance of bidirectional context in image captioning, moving beyond the limitations of unidirectional models.
  • Compactness and Parallelism Win: The core insight is that a compact architecture that allows for parallel bidirectional decoding is more effective than sequential refinement models.
  • Implicit Regularization is Powerful: The architecture's inherent bidirectional nature acts as a strong implicit regularizer, often more impactful than explicit interaction mechanisms.
  • Sentence-Level Ensemble is Key: The strategy of selecting the best caption from the L2R and R2L outputs (sentence-level ensemble) is a significant contributor to performance gains.
  • Explicit Interaction is Overestimated: Contrary to some prior work, the authors find that explicit interaction between L2R and R2L flows is less critical than the overall bidirectional architecture and sentence-level ensemble.

Key Results

  • State-of-the-Art Performance: CBTrans achieves new state-of-the-art results on the MSCOCO benchmark, particularly in CIDEr scores, outperforming previous non-vision-language-pretraining models.
  • CIDEr Improvement: CBTrans achieves a significant improvement of about 5% in CIDEr score when used in an ensemble setting compared to other models.
  • Ablation Study Findings:
    • The compact bidirectional architecture and sentence-level ensemble play more important roles than explicit interaction.
  • Explicit interaction mechanisms (controlled by parameter λ) offer only slight improvements.
  • The sentence-level ensemble contributes more than 2% in CIDEr for CBTrans and over 4% for CBLSTM when combined with model ensembles.
  • Qualitative Improvements: The generated captions are generally more descriptive and accurate, often capturing nuances missed by unidirectional models.

Limitations and Future Work

  • Bad Endings in R2L: The paper notes that the R2L branch can sometimes produce undesirable "bad endings" after CIDEr score optimization, which can negatively impact BLEU scores. They suggest this can be mitigated by removing bad endings or incorporating BLEU into the scoring function.
  • Model Capacity: The authors suggest that the limited model capacity of CBLSTM might be why the regularization effect of the compact architecture is less pronounced compared to CBTrans.
  • Integration with Pre-training: The authors propose that their bidirectional decoder architecture could be integrated into vision-language pre-training frameworks to potentially enhance caption quality further.
  • Further Exploration of Ensemble: The paper hints at further exploration of combining word-level and sentence-level ensembles seamlessly.

Summary updated

Abstract

Most current image captioning models typically generate captions from left-to-right. This unidirectional property makes them can only leverage past context but not future context. Though refinement-based models can exploit both past and future context by generating a new caption in the second stage based on pre-retrieved or pre-generated captions in the first stage, the decoder of these models generally consists of two networks~(i.e. a retriever or captioner in the first stage and a captioner in the second stage), which can only be executed sequentially. In this paper, we introduce a Compact Bidirectional Transformer model for image captioning that can leverage bidirectional context implicitly and explicitly while the decoder can be executed parallelly. Specifically, it is implemented by tightly coupling left-to-right(L2R) and right-to-left(R2L) flows into a single compact model to serve as a regularization for implicitly exploiting bidirectional context and optionally allowing explicit interaction of the bidirectional flows, while the final caption is chosen from either L2R or R2L flow in a sentence-level ensemble manner. We conduct extensive ablation studies on MSCOCO benchmark and find that the compact bidirectional architecture and the sentence-level ensemble play more important roles than the explicit interaction mechanism. By combining with word-level ensemble seamlessly, the effect of sentence-level ensemble is further enlarged. We further extend the conventional one-flow self-critical training to the two-flows version under this architecture and achieve new state-of-the-art results in comparison with non-vision-language-pretraining models. Finally, we verify the generality of this compact bidirectional architecture by extending it to LSTM backbone. Source code is available at https://github.com/YuanEZhou/cbtic.
BibTeX
@misc{zhou2022image,
  author = {Yuanen Zhou and Zhenzhen Hu and Daqing Liu and Huixia Ben and Meng Wang and Hong, Richang and Wang, Meng},
  title = {{Image Captioning via Compact Bidirectional Architecture}},
  year = {2022},
  eprint = {2201.01984},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2201.01984},
}
Figure