TL;DR
This paper introduces CBTrans, a compact bidirectional architecture that generates image captions from both left-to-right and right-to-left simultaneously within a single, parameter-efficient model. By leveraging bidirectional context and a novel sentence-level ensemble mechanism, it achieves state-of-the-art performance on the MSCOCO benchmark without the sequential overhead of traditional multi-stage refinement models.
What Problem Does This Paper Address?
Most image captioning models are "unidirectional"—they generate sentences word-by-word from left to right. While effective, this approach has a fundamental flaw: when predicting the next word, the model can only look at the "past" (words already generated) and has no access to "future" context.
Imagine trying to describe a photo of a man on a beach. If you start with "A man...", your choice of the next word might depend on whether you plan to end the sentence with "...is walking a horse" or "...is surfing."
Previous attempts to solve this used "refinement-based" models that generate a draft caption and then "refine" it in a second stage. However, these are computationally expensive because they require two separate networks running one after the other. The authors of this paper sought a way to get the benefits of bidirectional context—knowing both the start and end of a sentence—while keeping the model compact and fast.
Methods
The core innovation is the Compact Bidirectional Transformer (CBTrans). Unlike models that use two separate networks for different directions, CBTrans integrates left-to-right (L2R) and right-to-left (R2L) "flows" into a single Transformer or LSTM network with shared parameters.
Key technical highlights include:
- Weight Sharing: L2R and R2L flows share the same weights, making the model "compact" and acting as a form of multi-task regularization.
- Parallel Decoding: Both flows are executed in parallel during inference, avoiding the sequential bottleneck of refinement models.
- Bidirectional Interaction: An optional module allows the L2R flow to explicitly attend to the R2L flow (and vice versa) during generation, effectively "peeking" into the future context.
- Sentence-Level Ensemble: After generating both L2R and R2L candidates, a ranker chooses the best one based on their probabilities.
- Two-Flow Self-Critical Training: The authors extended the standard Reinforcement Learning-based training (SCST) to optimize both directional flows simultaneously.
The authors also demonstrated the generality of this approach by applying it to an LSTM backbone, creating CBLSTM.
Datasets
The models were evaluated on the MSCOCO dataset, the gold standard for image captioning. The authors used the popular "Karpathy" splits (113k training, 5k validation, 5k test images) for offline experiments and also submitted results to the official MSCOCO online testing server.
Key Insights and Ideas
- Compactness as Regularization: One of the most surprising findings was that simply training a single model to perform both L2R and R2L tasks (with shared weights) significantly improved performance. The shared architecture forces the model to learn more robust features.
- Sentence-Level Ensemble is King: The authors found that the "sentence-level ensemble"—choosing the best output between the two flows—contributed more to the final score than the complex "explicit interaction" mechanism between the flows.
- Ensemble Synergy: When combined with traditional word-level ensembles (averaging multiple models), the sentence-level ensemble effect is magnified, leading to substantial gains.
- Implicit vs. Explicit Context: While explicit interaction helps, the implicit benefit of the bidirectional architecture (the regularization effect) is the primary driver of success.
Key Results
CBTrans achieved new state-of-the-art results among models that do not use massive vision-language pre-training:
- CIDEr Gains: On the MSCOCO test split, CBTrans improved the CIDEr score of a standard Transformer from 111.7 to 114.4 (using Up-Down features) and from 117.6 to 121.8 (using VinVL features).
- Ensemble Performance: In an ensemble setting, CBTrans reached a CIDEr score of 140.3, outperforming other competitive models like X-LAN and RSTNet by a significant margin (approx. 5%).
- Online Leaderboard: On the official MSCOCO online server, CBTrans achieved a CIDEr (c40) score of 138.6, making it one of the top-performing non-pretrained models.
Qualitative results show that CBTrans often "borrows" the best parts of both L2R and R2L predictions to form a more accurate final caption.
Limitations and Future Work
The authors noted a specific issue with the R2L flow: it sometimes produces "bad endings" (which are actually the beginnings of the original sentences) that sound unnatural. For example, it might generate "of a man with a soccer ball on a field" instead of a complete sentence. They mitigated this with a post-processing trick but suggest that better reward functions during training could solve it more elegantly.
Future work could involve integrating this compact bidirectional design into large-scale vision-language pre-training frameworks, potentially replacing their unidirectional decoders to further boost caption quality.
Summary updated