An Overview of Cross-media Retrieval: Concepts, Methodologies, Benchmarks and Challenges

Yuxin Peng, Xin Huang, Yunzhen Zhao

Abstract

Multimedia retrieval plays an indispensable role in big data utilization. Past efforts mainly focused on single-media retrieval. However, the requirements of users are highly flexible, such as retrieving relevant audio clips with an image query. Challenges stemming from the "media gap"—where representations of different media types are inconsistent—have attracted increasing attention. This paper reviews more than 100 references to provide a comprehensive overview of cross-media retrieval concepts, methodologies, and benchmarks. We introduce XMedia, the first publicly available dataset featuring five media types (text, image, video, audio, and 3D models), to help researchers evaluate their methods efficiently.

Introduction: Bridging the "Media Gap"

Traditional multimedia retrieval is typically "single-media" (e.g., searching for images using an image). However, modern user needs demand cross-modal flexibility. The fundamental hurdle is the "media gap": different media types have inconsistent mathematical representations. An image is a grid of pixels, while a text description is a sequence of words; they exist in entirely different feature spaces, making direct similarity measurements nearly impossible.

The general framework of cross-media retrieval, illustrating the flow from queries to results across different media types.

Methodological Taxonomy

The paper categorizes cross-media retrieval into two primary technical lineages:

1. Common Space Learning

This is the mainstream approach, aiming to project different media types into a shared "intermediate" space where features can be compared directly. Key sub-categories include:

  • Traditional Statistical Analysis: Using techniques like Canonical Correlation Analysis (CCA) to find linear correlations.
  • DNN-based Methods: Leveraging Deep Neural Networks to learn complex, non-linear mappings.
  • Cross-media Hashing: Mapping data into a common Hamming space (binary strings) to accelerate retrieval speeds.
  • Metric Learning & Learning to Rank: Optimizing the space to ensure similar items are "close" and dissimilar items are "far."

2. Cross-media Similarity Measurement

Instead of building a common space, these methods compute similarities directly by analyzing known relationships (like co-occurrence on a webpage) or using graph-based models to propagate similarity through a network of data points.

A detailed taxonomy of representative works in cross-media retrieval, categorized by methodology.

Benchmarking with the XMedia Dataset

While standard datasets like Wikipedia (image/text) and NUS-WIDE exist, they often only cover two media types. To address this, the authors introduced XMedia. It consists of 12,000 media instances across 20 categories (e.g., "insect," "tiger," "elephant"). Crucially, it includes five media types: text, image, video, audio, and 3D models. This allows researchers to test "all-to-all" retrieval scenarios.

Key Insights and Results

Benchmarking results reveal several critical trends in the field:

  • The Power of Supervision: "Fully supervised" methods—those using semantic category labels during training—consistently outperform unsupervised methods relying solely on raw data correlations.
  • Non-linearity is Essential: While traditional linear methods (like CCA) are efficient, DNN-based approaches are significantly better at capturing the abstraction needed to bridge the media gap.
  • Deep Learning & Graphs Win: Methods like Joint Representation Learning (JRL) and the authors' unified hypergraph method ($S^2UPG$) achieved the highest Mean Average Precision (MAP) scores. On the XMedia dataset, $S^2UPG$ achieved a MAP of 0.371, doubling the performance of traditional CCA (0.187).
  • Feature Impact: Transitioning from traditional features (like BoVW) to CNN-based features (like AlexNet) provided a massive boost in retrieval accuracy across all underlying algorithms.

Future Directions

The authors identify several open challenges for the next generation of retrieval systems:

  • Scaling Up: Current datasets are small compared to the open web. Future work requires larger, noisier datasets like the proposed XMediaNet (targeting 100,000+ instances).
  • Contextual Awareness: Most models ignore "context correlation," such as the metadata or surrounding text found on a webpage that links disparate media types.
  • End-to-End Architectures: There is a growing need for systems that process raw media directly into retrieval results, rather than relying on pre-extracted features.

Summary updated

Abstract

Multimedia retrieval plays an indispensable role in big data utilization. Past efforts mainly focused on single-media retrieval. However, the requirements of users are highly flexible, such as retrieving the relevant audio clips with one query of image. So challenges stemming from the "media gap", which means that representations of different media types are inconsistent, have attracted increasing attention. Cross-media retrieval is designed for the scenarios where the queries and retrieval results are of different media types. As a relatively new research topic, its concepts, methodologies and benchmarks are still not clear in the literatures. To address these issues, we review more than 100 references, give an overview including the concepts, methodologies, major challenges and open issues, as well as build up the benchmarks including datasets and experimental results. Researchers can directly adopt the benchmarks to promptly evaluate their proposed methods. This will help them to focus on algorithm design, rather than the time-consuming compared methods and results. It is noted that we have constructed a new dataset XMedia, which is the first publicly available dataset with up to five media types (text, image, video, audio and 3D model). We believe this overview will attract more researchers to focus on cross-media retrieval and be helpful to them.
BibTeX
@article{peng2018overview,
  author = {Yuxin Peng and Xin Huang and Yunzhen Zhao},
  title = {{An Overview of Cross-media Retrieval: Concepts, Methodologies, Benchmarks and Challenges}},
  journal = {IEEE Trans. Circuits Syst. Video Technol.},
  year = {2018},
  url = {https://openreview.net/forum?id=W8fnb3_uEjH},
}
Figure