Image Captioning via Compact Bidirectional Architecture

Zijie Song, Yuanen Zhou, Zhenzhen Hu, Daqing Liu, Huixia Ben, Richang Hong, Meng Wang

TL;DR

This paper introduces CBTrans, a compact bidirectional architecture that generates image captions from both left-to-right and right-to-left simultaneously within a single, parameter-efficient model. By leveraging bidirectional context and a novel sentence-level ensemble mechanism, it achieves state-of-the-art performance on the MSCOCO benchmark without the sequential overhead of traditional multi-stage refinement models.

What Problem Does This Paper Address?

Most image captioning models are "unidirectional"—they generate sentences word-by-word from left to right. While effective, this approach has a fundamental flaw: when predicting the next word, the model can only look at the "past" (words already generated) and has no access to "future" context.

Imagine trying to describe a photo of a man on a beach. If you start with "A man...", your choice of the next word might depend on whether you plan to end the sentence with "...is walking a horse" or "...is surfing."

Previous attempts to solve this used "refinement-based" models that generate a draft caption and then "refine" it in a second stage. However, these are computationally expensive because they require two separate networks running one after the other. The authors of this paper sought a way to get the benefits of bidirectional context—knowing both the start and end of a sentence—while keeping the model compact and fast.

A conceptual overview comparing unidirectional, refinement-based, and the proposed compact bidirectional generation.

Methods

The core innovation is the Compact Bidirectional Transformer (CBTrans). Unlike models that use two separate networks for different directions, CBTrans integrates left-to-right (L2R) and right-to-left (R2L) "flows" into a single Transformer or LSTM network with shared parameters.

Key technical highlights include:

  • Weight Sharing: L2R and R2L flows share the same weights, making the model "compact" and acting as a form of multi-task regularization.
  • Parallel Decoding: Both flows are executed in parallel during inference, avoiding the sequential bottleneck of refinement models.
  • Bidirectional Interaction: An optional module allows the L2R flow to explicitly attend to the R2L flow (and vice versa) during generation, effectively "peeking" into the future context.
  • Sentence-Level Ensemble: After generating both L2R and R2L candidates, a ranker chooses the best one based on their probabilities.
  • Two-Flow Self-Critical Training: The authors extended the standard Reinforcement Learning-based training (SCST) to optimize both directional flows simultaneously.
The architecture of CBTrans, showing how L2R and R2L flows are coupled within the decoder.

The authors also demonstrated the generality of this approach by applying it to an LSTM backbone, creating CBLSTM.

Fig. 3. Illustration of Compact Bidirectional LSTM model (CBLSTM) for Image Captioning. Best view in color.

Datasets

The models were evaluated on the MSCOCO dataset, the gold standard for image captioning. The authors used the popular "Karpathy" splits (113k training, 5k validation, 5k test images) for offline experiments and also submitted results to the official MSCOCO online testing server.

Key Insights and Ideas

  • Compactness as Regularization: One of the most surprising findings was that simply training a single model to perform both L2R and R2L tasks (with shared weights) significantly improved performance. The shared architecture forces the model to learn more robust features.
  • Sentence-Level Ensemble is King: The authors found that the "sentence-level ensemble"—choosing the best output between the two flows—contributed more to the final score than the complex "explicit interaction" mechanism between the flows.
  • Ensemble Synergy: When combined with traditional word-level ensembles (averaging multiple models), the sentence-level ensemble effect is magnified, leading to substantial gains.
  • Implicit vs. Explicit Context: While explicit interaction helps, the implicit benefit of the bidirectional architecture (the regularization effect) is the primary driver of success.

Key Results

CBTrans achieved new state-of-the-art results among models that do not use massive vision-language pre-training:

  • CIDEr Gains: On the MSCOCO test split, CBTrans improved the CIDEr score of a standard Transformer from 111.7 to 114.4 (using Up-Down features) and from 117.6 to 121.8 (using VinVL features).
  • Ensemble Performance: In an ensemble setting, CBTrans reached a CIDEr score of 140.3, outperforming other competitive models like X-LAN and RSTNet by a significant margin (approx. 5%).
  • Online Leaderboard: On the official MSCOCO online server, CBTrans achieved a CIDEr (c40) score of 138.6, making it one of the top-performing non-pretrained models.

Qualitative results show that CBTrans often "borrows" the best parts of both L2R and R2L predictions to form a more accurate final caption.

Qualitative examples comparing CBTrans outputs with unidirectional baselines.

Limitations and Future Work

The authors noted a specific issue with the R2L flow: it sometimes produces "bad endings" (which are actually the beginnings of the original sentences) that sound unnatural. For example, it might generate "of a man with a soccer ball on a field" instead of a complete sentence. They mitigated this with a post-processing trick but suggest that better reward functions during training could solve it more elegantly.

Future work could involve integrating this compact bidirectional design into large-scale vision-language pre-training frameworks, potentially replacing their unidirectional decoders to further boost caption quality.

Summary updated

Abstract

Most current image captioning models typically generate captions from left-to-right. This unidirectional property makes them can only leverage past context but not future context. Though refinement-based models can exploit both past and future context by generating a new caption in the second stage based on pre-retrieved or pre-generated captions in the first stage, the decoder of these models generally consists of two networks~(i.e. a retriever or captioner in the first stage and a captioner in the second stage), which can only be executed sequentially. In this paper, we introduce a Compact Bidirectional Transformer model for image captioning that can leverage bidirectional context implicitly and explicitly while the decoder can be executed parallelly. Specifically, it is implemented by tightly coupling left-to-right(L2R) and right-to-left(R2L) flows into a single compact model to serve as a regularization for implicitly exploiting bidirectional context and optionally allowing explicit interaction of the bidirectional flows, while the final caption is chosen from either L2R or R2L flow in a sentence-level ensemble manner. We conduct extensive ablation studies on MSCOCO benchmark and find that the compact bidirectional architecture and the sentence-level ensemble play more important roles than the explicit interaction mechanism. By combining with word-level ensemble seamlessly, the effect of sentence-level ensemble is further enlarged. We further extend the conventional one-flow self-critical training to the two-flows version under this architecture and achieve new state-of-the-art results in comparison with non-vision-language-pretraining models. Finally, we verify the generality of this compact bidirectional architecture by extending it to LSTM backbone. Source code is available at https://github.com/YuanEZhou/cbtic.
BibTeX
@misc{song2022image,
  author = {Zijie Song and Yuanen Zhou and Zhenzhen Hu and Daqing Liu and Huixia Ben and Richang Hong and Meng Wang},
  title = {{Image Captioning via Compact Bidirectional Architecture}},
  year = {2022},
  eprint = {2201.01984},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2201.01984},
}
Figure