An Overview of Cross-Media Retrieval: Concepts, Methodologies, Benchmarks, and Challenges

Yuxin Peng, Xin Huang, Yunzhen Zhao

TL;DR

This paper provides a comprehensive roadmap for cross-media retrieval—the ability to search across different formats, such as using an image to find a related audio clip. The authors categorize over 100 research efforts, establish a standardized benchmarking framework, and introduce XMedia, the first publicly available dataset to encompass five distinct media types (text, image, video, audio, and 3D models).

What Problem Does This Paper Address?

Traditional multimedia retrieval is usually "single-media" (e.g., searching for images using an image). However, modern users need more flexibility. The fundamental hurdle is the "media gap": different media types have inconsistent representations. An image is a grid of pixels, while a text description is a sequence of words; they exist in entirely different mathematical feature spaces, making direct similarity measurements nearly impossible.

The authors tackle the lack of a unified framework in this field, addressing the inconsistent terminology, the absence of diverse datasets, and the difficulty of comparing different algorithmic approaches.

Methods

The paper categorizes cross-media retrieval into two primary technical lineages:

  1. Common Space Learning (The Mainstream): These methods aim to project different media types into a shared "intermediate" space where their features can be compared directly. Sub-categories include:
  2. Traditional Statistical Analysis: Using techniques like Canonical Correlation Analysis (CCA) to find linear correlations.
  3. DNN-based Methods: Leveraging Deep Neural Networks to learn complex, non-linear mappings between media types.
  4. Cross-media Hashing: Mapping data into a common Hamming space (binary strings) to significantly accelerate retrieval speeds.
  5. Metric Learning & Learning to Rank: Optimizing the space specifically to ensure similar items are "close" and dissimilar items are "far."
  6. Cross-media Similarity Measurement: Instead of building a common space, these methods compute similarities directly by analyzing known relationships (like co-occurrence on a webpage) or using graph-based models to propagate similarity through a network of data points.

Datasets

The authors review several standard datasets like Wikipedia (image/text) and NUS-WIDE (image/tags). However, they highlight a major limitation: most existing sets only cover two media types.

To solve this, they introduced XMedia. It consists of 12,000 media instances across 20 categories (e.g., "insect," "tiger," "elephant"). Crucially, it includes five media types: text, image, video, audio, and 3D models. This allows researchers to test "all-to-all" retrieval rather than just "image-to-text."

Key Insights and Ideas

  • The Power of Supervision: The paper demonstrates that "fully supervised" methods—those that use semantic category labels during training—consistently outperform unsupervised methods that rely solely on raw data correlations.
  • Non-linearity is Essential: While traditional linear methods (like CCA) are efficient, they struggle with real-world data. DNN-based approaches are better at capturing the "abstraction" needed to bridge the media gap.
  • Fine-Grained Modeling: The authors' own work ($S^2UPG$) suggests that breaking media into "patches" or segments (fine-grained information) helps the model understand local correlations that global features might miss.
  • The Efficiency-Accuracy Trade-off: Hashing is the gold standard for speed, but it often comes at the cost of retrieval precision. Finding the "sweet spot" remains a primary research focus.

Key Results

The benchmarking results show a clear hierarchy in performance:

  • Deep Learning & Graphs Win: Methods like JRL (Joint Representation Learning) and $S^2UPG$ (the authors' unified hypergraph method) generally achieved the highest Mean Average Precision (MAP) scores across datasets.
  • Feature Impact: Moving from traditional features (like BoVW) to CNN-based features (like AlexNet) provided a massive boost in retrieval accuracy, regardless of the underlying retrieval algorithm.
  • XMedia Performance: On the XMedia dataset, the $S^2UPG$ method achieved a MAP of 0.371, significantly outperforming traditional CCA (0.187).

Limitations and Future Work

  • Dataset Scale: Even with XMedia, the authors acknowledge that current datasets are small compared to the "big data" found on the open web. They suggest the need for even larger, noisier datasets like their proposed XMediaNet (targeting 100,000+ instances).
  • Contextual Neglect: Most current models ignore "context correlation," such as the fact that an image and an audio clip found on the same webpage are likely related even if their raw features don't match.
  • End-to-End Architectures: There is a call for more "end-to-end" deep learning systems that take raw media as input and output retrieval results directly, rather than relying on pre-extracted features.

Summary updated

Abstract

Multimedia retrieval plays an indispensable role in big data utilization. Past efforts mainly focused on single-media retrieval. However, the requirements of users are highly flexible, such as retrieving the relevant audio clips with one query of image. So challenges stemming from the “media gap,” which means that representations of different media types are inconsistent, have attracted increasing attention. Cross-media retrieval is designed for the scenarios where the queries and retrieval results are of different media types. As a relatively new research topic, its concepts, methodologies, and benchmarks are still not clear in the literature. To address these issues, we review more than 100 references, give an overview including the concepts, methodologies, major challenges, and open issues, as well as build up the benchmarks, including data sets and experimental results. Researchers can directly adopt the benchmarks to promptly evaluate their proposed methods. This will help them to focus on algorithm design, rather than the time-consuming compared methods and results. It is noted that we have constructed a new data set <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">XMedia</i> , which is the first publicly available data set with up to five media types (text, image, video, audio, and 3-D model). We believe this overview will attract more researchers to focus on cross-media retrieval and be helpful to them.
BibTeX
@article{peng2018overview,
  author = {Yuxin Peng and Xin Huang and Yunzhen Zhao},
  title = {{An Overview of Cross-Media Retrieval: Concepts, Methodologies, Benchmarks, and Challenges}},
  journal = {IEEE Trans. Circuits Syst. Video Technol.},
  year = {2018},
  url = {https://openreview.net/forum?id=W8fnb3_uEjH},
}

Loading related content...

Figure