Cross-Media Shared Representation by Hierarchical Learning with Multiple Deep Networks

Yuxin Peng, Xin Huang, Jinwei Qi

Introduction: Bridging the Heterogeneity Gap

In our multi-modal digital landscape, the ability to search across different media types—such as using an image to find related text or vice versa—is increasingly vital. This task, known as cross-media retrieval, faces a fundamental challenge: the "heterogeneity gap." Images (grids of pixels) and text (sequences of words) exist in entirely different mathematical spaces, making direct comparison difficult.

While Deep Neural Networks (DNNs) have been employed to bridge this gap, existing approaches often suffer from two major limitations. First, they typically model media types in isolation initially, ignoring early-stage correlations. Second, the integration into a "shared" space is often handled by shallow networks that fail to capture complex, non-linear relationships between visual and semantic features.

The CMDN Framework

To address these flaws, the authors propose the Cross-Media Multiple Deep Network (CMDN) framework. CMDN utilizes a two-stage hierarchical learning process to create a robust shared representation that captures both internal characteristics and cross-modal relationships.

The two-stage hierarchical learning architecture of CMDN, illustrating the separation of intra-media and inter-media representation learning.

Stage 1: Complementary Separate Representation

Unlike traditional methods, CMDN learns two distinct types of representations for each media type simultaneously:

  • Intra-media Representation: Uses Stacked Autoencoders (SAE) to extract high-level semantic features from within a single media type (e.g., identifying shapes or textures in an image).
  • Inter-media Representation: Employs a Multimodal Deep Belief Network (DBN) to capture the correlations between media types, using the relationship between modalities as a feature itself.

Stage 2: Hierarchical Shared Representation

The model then merges these complementary representations through a deeper, two-level strategy. First, a joint Restricted Boltzmann Machine (RBM) combines the intra- and inter-media features for each modality. Second, a Stacked Bimodal Autoencoder refines these into a final, unified shared representation space. This stacking allows the model to learn significantly more complex mappings than flat, single-layer architectures.

Experimental Evaluation

The CMDN model was evaluated on three benchmark datasets: Wikipedia (2,866 pairs), NUS-WIDE-10k (10,000 pairs), and Pascal Sentences (1,000 pairs). Performance was measured primarily using Mean Average Precision (MAP).

Precision-Recall curves for image-to-text (left) and text-to-image (right) retrieval on the Wikipedia dataset, showing CMDN's superior performance over baseline methods.

Key Results

CMDN consistently outperformed seven state-of-the-art methods, including CCA, JRL, and Bimodal AE. Notable MAP improvements include:

  • Wikipedia: Achieved 0.359, compared to the previous best of 0.311 (JRL).
  • NUS-WIDE-10k: Achieved 0.374, a significant jump from JRL's 0.294.
  • Pascal Sentences: Achieved 0.334, outperforming JRL at 0.293.

Qualitative analysis via Precision-Recall curves demonstrates that CMDN maintains higher precision even as recall increases, indicating a more stable and accurate retrieval process across both image-to-text and text-to-image tasks.

Conclusion and Future Directions

The success of CMDN highlights the importance of treating inter-media correlation as an input feature rather than just a final goal. By preserving the unique characteristics of each media type through intra-media modeling while simultaneously capturing cross-modal links, the hierarchical approach effectively bridges the heterogeneity gap.

Future work may involve extending the framework to other media types like audio and video, exploring semi-supervised learning to leverage unlabeled data, and integrating more advanced architectures like Convolutional Neural Networks (CNNs) for visual feature extraction.

Summary updated

BibTeX
@inproceedings{peng2016cross,
  author = {Yuxin Peng and Xin Huang and Jinwei Qi},
  title = {{Cross-Media Shared Representation by Hierarchical Learning with Multiple Deep Networks}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2016},
}
Figure