Cross-modal Bidirectional Translation via Reinforcement Learning

Jinwei Qi, Yuxin Peng

Introduction

In the field of artificial intelligence, cross-modal retrieval—the ability to search for images using text queries or vice versa—remains a significant challenge due to the modality gap. Images and text are fundamentally different: one is represented by continuous pixel values, while the other consists of discrete linguistic units. Traditional approaches typically attempt to bridge this gap by projecting both modalities into a shared "common space." However, these unidirectional projections often fail to capture the complex, intrinsic structures of each data type.

Researchers from Peking University have proposed a novel framework called Cross-modal Bidirectional Translation (CBT). This approach shifts the paradigm from simple mapping to direct translation, treating images and text as two distinct languages. By leveraging reinforcement learning (RL) to play a "bidirectional game," the model learns to translate between modalities while preserving the core semantic information of the original data.

Methodology

The CBT framework is built upon two specialized agents: one for visual processing and one for textual processing.

1. The Translation Mechanism

  • Image Encoding: Images are processed through a VGGNet. Rather than using a global feature vector, the model treats different image regions as a sequence, employing an LSTM to capture spatial context and fine-grained local information.
  • Text Encoding: Textual data is handled via a Word CNN and an LSTM to capture temporal and linguistic dependencies.
  • Bidirectional Pathways: The architecture features two primary translation pathways: Image $\rightarrow$ Text and Text $\rightarrow$ Image. These pathways are unified by a shared middle layer governed by semantic constraints (softmax loss) to ensure alignment with categorical labels.
The CBT framework architecture, illustrating the bidirectional translation pathways between text and image agents with semantic constraints.

2. Reinforcement Learning Strategy

Because there are no "perfect" ground-truth translations in a continuous feature space, the authors frame the training as an RL problem using Policy Gradient methods. The model aims to maximize a dual reward function:

  • Inter-modality Reward ($r^{inter}$): Measures the similarity between the translated representation and the actual target representation in the opposite modality.
  • Intra-modality Reward ($r^{intra}$): Measures the success of a "round-trip" translation (e.g., Image $\rightarrow$ Text $\rightarrow$ Image). This reconstruction reward ensures that the translation process preserves the original identity of the data.

Experimental Evaluation

The CBT model was evaluated on three benchmark datasets:

  1. Wikipedia Dataset: 2,866 image/text pairs across 10 categories.
  2. Pascal Sentence Dataset: 1,000 images, each paired with 5 independent sentences, across 20 categories.
  3. XMediaNet: A large-scale dataset featuring 40,000 image/text pairs and 200 categories.

Results and Discussion

The CBT model demonstrated superior performance compared to 11 state-of-the-art methods, including CCA, Deep-SM, and CCL. Performance was quantified using Mean Average Precision (MAP):

  • Wikipedia: CBT achieved a MAP of 0.490 (vs. 0.481 for CCL).
  • Pascal Sentence: CBT reached 0.592 (vs. 0.569 for CCL).
  • XMediaNet: On the large-scale benchmark, CBT hit 0.576 (vs. 0.533 for CCL).

Ablation studies highlighted the importance of balancing the reward signals. As shown in the sensitivity analysis, the model generally performs best when the reward is weighted more heavily toward inter-modality correlation (typically where $\alpha > 0.5$), though the intra-modality reconstruction reward remains critical for achieving peak accuracy.

Performance (Average MAP) across the Wikipedia, Pascal Sentence, and XMediaNet datasets as a function of the reward weighting parameter α.

Conclusion

The CBT framework demonstrates that treating cross-modal retrieval as a bidirectional translation task—rather than a simple projection—yields more robust semantic alignments. By incorporating reinforcement learning and reconstruction rewards, the model effectively bridges the modality gap. Future work may explore unsupervised learning techniques to leverage the vast amounts of unlabeled multimodal data available online, potentially reducing the reliance on supervised category labels.

Summary updated

Abstract

The inconsistent distribution and representation of image and text make it quite challenging to measure their similarity, and construct correlation between them. Inspired by neural machine translation to establish a corresponding relationship between two entirely different languages, we attempt to treat images as a special kind of language to provide visual descriptions, so that translation can be conduct between bilingual pair of image and text to effectively explore cross-modal correlation. Thus, we propose Cross-modal Bidirectional Translation (CBT) approach, and further explore the utilization of reinforcement learning to improve the translation process. First, a cross-modal translation mechanism is proposed, where image and text are treated as bilingual pairs, and cross-modal correlation can be effectively captured in both feature spaces of image and text by bidirectional translation training. Second, cross-modal reinforcement learning is proposed to perform a bidirectional game between image and text, which is played as a round to promote the bidirectional translation process. Besides, both inter-modality and intra-modality reward signals can be extracted to provide complementary clues for boosting cross-modal correlation learning. Experiments are conducted to verify the performance of our proposed approach on cross-modal retrieval, compared with 11 state-of-the-art methods on 3 datasets.
BibTeX
@inproceedings{qi2018cross,
  author = {Jinwei Qi and Yuxin Peng},
  title = {{Cross-modal Bidirectional Translation via Reinforcement Learning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.24963/ijcai.2018/365},
}
Figure