Abstract: Most current image captioning models typically generate captions from left to right. This unidirectional property makes them can only leverage past context but not future context. Though recent refinement-based models can exploit both past and future context by generating a new caption in the second stage based on pre-retrieved or pre-generated captions in the first stage, the decoder of these models generally consists of two networks (i.e. a retriever or captioner in the first stage and a refiner in the second stage), which can only be executed sequentially. In this paper, we introduce a Compact Bidirectional Transformer model for image captioning that can leverage bidirectional context implicitly and explicitly while the decoder can be executed parallelly. Specifically, it is implemented by tightly coupling left-to-right (L2R) and right-to-left (R2L) flows into a single compact model (i.e. implicitly) and optionally allowing interaction of the two flows (i.e. explicitly), while the final caption is chosen from either L2R or R2L flow in a sentence-level ensemble manner. We conduct extensive ablation studies on the MSCOCO benchmark and find that the compact architecture, which serves as a regularization for implicitly exploiting bidirectional context, and the sentence-level ensemble play more important roles than the explicit interaction mechanism. By combining with word-level ensemble seamlessly, the effect of the sentence-level ensemble is further enlarged. We further extend the conventional one-flow self-critical training to the two-flows version under this architecture and achieve new state-of-the-art results in comparison with non-vision-language-pretraining models.
Introduction
Standard image captioning models are typically "unidirectional," generating words sequentially from left to right. This design limitation means that when predicting a word, the model only has access to the "past" context, remaining blind to the "future" context. While refinement-based models attempt to solve this by using a second network to "look back" at a draft caption, these systems are often computationally expensive, requiring sequential passes and doubling parameter counts.
To address these inefficiencies, we propose the Compact Bidirectional Transformer (CBTrans). This architecture exploits bidirectional context within a single, parameter-efficient, and parallelizable model by integrating forward and backward generation flows.
Methodology
The core innovation of CBTrans lies in its compact architecture that shares parameters between left-to-right (L2R) and right-to-left (R2L) flows.
Key Components:
- Compact Architecture: Instead of separate models, CBTrans uses a single Transformer decoder. It distinguishes directions using special prefix tokens:
<l2r>and<r2l>. This shared parameter space acts as a powerful regularizer, forcing the model to learn more robust language representations. - Bidirectional Interactive Attention (Optional): This mechanism allows the L2R flow to explicitly "peek" at the R2L flow’s hidden states during generation, providing direct access to future context.
- Sentence-Level Ensemble: During inference, the model generates both a forward and a backward caption. It ranks them based on log-probabilities and selects the superior candidate.
- Two-Flow Self-Critical Training: We extended standard Reinforcement Learning-based training (SCST) to optimize both flows simultaneously using the CIDEr metric as a reward.
Experiments and Results
We evaluated CBTrans on the MSCOCO benchmark using Karpathy splits and the official online testing server. The model was tested with both standard Up-Down features and high-performance VinVL features.
As shown in the results, CBTrans achieved new state-of-the-art performance among models that do not utilize massive vision-language pre-training. On the MSCOCO Karpathy test split, CBTrans reached a CIDEr score of 140.3 (using VinVL features and ensemble), a significant improvement over standard Transformer baselines which typically range between 127 and 132.
Key Insights:
- Regularization over Interaction: Surprisingly, the compact architecture itself provides more benefit than the explicit interaction mechanism. Training on both directions simultaneously improves the performance of individual flows.
- Efficiency: As a single-model architecture, CBTrans is more parameter-efficient than multi-stage refinement models. Furthermore, the directional flows can be executed in parallel on a GPU, maintaining high inference speeds.
- Ensemble Synergy: The sentence-level ensemble effectively filters out directional quirks (such as awkward R2L endings) by selecting the most descriptive and grammatically sound version from the two flows.
Conclusion
CBTrans demonstrates that bidirectional context can be effectively captured through a compact, shared-parameter architecture. By combining implicit regularization with a sentence-level ensemble, the model achieves superior accuracy and efficiency compared to traditional unidirectional and refinement-based approaches. Future work will explore integrating this bidirectional logic into large-scale vision-language pre-training frameworks.
Source Code: https://github.com/YuanEZhou/CBTrans
Summary updated