Abstract
Cross-media retrieval is to return the results of various media types corresponding to the query of any media type. Existing researches generally focus on coarse-grained cross-media retrieval. When users submit an image of "Slaty-backed Gull" as a query, coarse-grained cross-media retrieval treats it as "Bird", so that users can only get the results of "Bird", which may include other bird species with similar appearance (image and video), descriptions (text) or sounds (audio), such as "Herring Gull". Such coarse-grained cross-media retrieval is not consistent with human lifestyle, where we generally have the fine-grained requirement of returning the exactly relevant results of "Slaty-backed Gull" instead of "Herring Gull". However, few researches focus on fine-grained cross-media retrieval, which is a highly challenging and practical task. Therefore, in this paper, we first construct a new benchmark for fine-grained cross-media retrieval, which consists of 200 fine-grained subcategories of the "Bird", and contains 4 media types, including image, text, video and audio. To the best of our knowledge, it is the first benchmark with 4 media types for fine-grained cross-media retrieval. Then, we propose a uniform deep model, namely FGCrossNet, which simultaneously learns 4 types of media without discriminative treatments. We jointly consider three constraints for better common representation learning: classification constraint ensures the learning of discriminative features, center constraint ensures the compactness characteristic of the features of the same subcategory, and ranking constraint ensures the sparsity characteristic of the features of different subcategories. Extensive experiments verify the usefulness of the new benchmark and the effectiveness of our FGCrossNet.
Introduction
Traditional cross-media retrieval systems typically operate at a coarse-grained level. If a user searches for a specific species, such as a "Slaty-backed Gull," a coarse-grained system might return any bird because it only understands the high-level concept of "Bird." This mismatch between system capabilities and human search intent highlights the need for fine-grained retrieval.
Fine-grained cross-media retrieval faces three primary hurdles: a lack of comprehensive datasets covering multiple modalities, the "heterogeneity gap" caused by differing data distributions across media types, and small inter-class variance between closely related subcategories. This paper addresses these challenges by introducing a novel benchmark and a uniform deep learning architecture.
The FGCrossNet Architecture
To bridge the heterogeneity gap, we propose FGCrossNet, a uniform deep model designed to standardize the input format for all media types. Instead of employing distinct networks for different modalities (e.g., LSTMs for text and CNNs for images), FGCrossNet utilizes a modified ResNet50 backbone for all inputs.
Uniform Representation through Preprocessing
The model achieves uniformity by converting non-visual data into 2D signals:
- Video: Represented by 25 uniformly spaced frames.
- Audio: Converted into 2D spectrograms, allowing the model to treat sound as a visual pattern.
- Text: Transformed into an image-like matrix using one-hot encoding followed by convolutional layers.
Triple-Constraint Loss Function
To handle the small inter-class variance inherent in fine-grained tasks, FGCrossNet is trained with three simultaneous constraints:
- Classification Constraint: Ensures the model identifies specific subcategories.
- Center Constraint: Pulls features of the same subcategory closer together (compactness).
- Ranking (Quadruplet) Constraint: Pushes features of different subcategories further apart (sparsity).
The Fine-Grained Cross-Media Benchmark
We introduce the first benchmark for fine-grained cross-media retrieval that includes four media types. Based on the CUB-200-2011 taxonomy, the dataset consists of 200 bird subcategories and includes 11,788 images, 8,000 text descriptions, 18,350 videos, and 12,000 audio clips.
Key Results and Analysis
FGCrossNet significantly outperforms existing state-of-the-art (SOTA) methods. In bi-modality retrieval tasks, FGCrossNet achieved a Mean Average Precision (MAP) of 0.366, nearly doubling the performance of the previous best method, MHTN (0.204). In multi-modality retrieval, it achieved a MAP of 0.412.
Ablation studies highlight the critical role of the specialized loss functions. Removing the Center and Ranking constraints caused the MAP to drop from 0.366 to 0.316, proving that classification alone is insufficient for fine-grained accuracy.
Conclusion
By shifting the focus to fine-grained retrieval and providing both a comprehensive benchmark and a uniform architecture, this work addresses a significant gap in cross-media retrieval. Future work will explore knowledge transfer between modalities and extending these techniques to automated reasoning tasks.
Summary updated