Abstract
Multimedia retrieval plays an indispensable role in big data utilization. Past efforts mainly focused on single-media retrieval. However, the requirements of users are highly flexible, such as retrieving relevant audio clips with an image query. Challenges stemming from the "media gap"—where representations of different media types are inconsistent—have attracted increasing attention. This paper reviews more than 100 references to provide a comprehensive overview of cross-media retrieval concepts, methodologies, and benchmarks. We introduce XMedia, the first publicly available dataset featuring five media types (text, image, video, audio, and 3D models), to help researchers evaluate their methods efficiently.
Introduction: Bridging the "Media Gap"
Traditional multimedia retrieval is typically "single-media" (e.g., searching for images using an image). However, modern user needs demand cross-modal flexibility. The fundamental hurdle is the "media gap": different media types have inconsistent mathematical representations. An image is a grid of pixels, while a text description is a sequence of words; they exist in entirely different feature spaces, making direct similarity measurements nearly impossible.
Methodological Taxonomy
The paper categorizes cross-media retrieval into two primary technical lineages:
1. Common Space Learning
This is the mainstream approach, aiming to project different media types into a shared "intermediate" space where features can be compared directly. Key sub-categories include:
- Traditional Statistical Analysis: Using techniques like Canonical Correlation Analysis (CCA) to find linear correlations.
- DNN-based Methods: Leveraging Deep Neural Networks to learn complex, non-linear mappings.
- Cross-media Hashing: Mapping data into a common Hamming space (binary strings) to accelerate retrieval speeds.
- Metric Learning & Learning to Rank: Optimizing the space to ensure similar items are "close" and dissimilar items are "far."
2. Cross-media Similarity Measurement
Instead of building a common space, these methods compute similarities directly by analyzing known relationships (like co-occurrence on a webpage) or using graph-based models to propagate similarity through a network of data points.
Benchmarking with the XMedia Dataset
While standard datasets like Wikipedia (image/text) and NUS-WIDE exist, they often only cover two media types. To address this, the authors introduced XMedia. It consists of 12,000 media instances across 20 categories (e.g., "insect," "tiger," "elephant"). Crucially, it includes five media types: text, image, video, audio, and 3D models. This allows researchers to test "all-to-all" retrieval scenarios.
Key Insights and Results
Benchmarking results reveal several critical trends in the field:
- The Power of Supervision: "Fully supervised" methods—those using semantic category labels during training—consistently outperform unsupervised methods relying solely on raw data correlations.
- Non-linearity is Essential: While traditional linear methods (like CCA) are efficient, DNN-based approaches are significantly better at capturing the abstraction needed to bridge the media gap.
- Deep Learning & Graphs Win: Methods like Joint Representation Learning (JRL) and the authors' unified hypergraph method ($S^2UPG$) achieved the highest Mean Average Precision (MAP) scores. On the XMedia dataset, $S^2UPG$ achieved a MAP of 0.371, doubling the performance of traditional CCA (0.187).
- Feature Impact: Transitioning from traditional features (like BoVW) to CNN-based features (like AlexNet) provided a massive boost in retrieval accuracy across all underlying algorithms.
Future Directions
The authors identify several open challenges for the next generation of retrieval systems:
- Scaling Up: Current datasets are small compared to the open web. Future work requires larger, noisier datasets like the proposed XMediaNet (targeting 100,000+ instances).
- Contextual Awareness: Most models ignore "context correlation," such as the metadata or surrounding text found on a webpage that links disparate media types.
- End-to-End Architectures: There is a growing need for systems that process raw media directly into retrieval results, rather than relying on pre-extracted features.
Summary updated