Mixer: Image to Multi-Modal Retrieval Learning for Industrial Application

Zida Cheng, Shuai Xiao, Zhonghua Zhai, Xiaoyi Zeng, Weilin HuangarXiv 2023 · DOI · arXiv

Introduction

In large-scale industrial search engines like Taobao, users frequently utilize image queries to find products. While image-to-image retrieval is a mature field, real-world product listings are multi-modal, containing both high-resolution images and descriptive text. This task, known as Image-to-Multi-Modal Retrieval (IMMR), presents three significant challenges: data noise in user click logs, information inequality between visual and textual modalities, and the immense scale of industrial datasets containing millions of categories.

To address these hurdles, researchers from Alibaba and Shanghai Jiao Tong University have introduced OCLEAR (Organizing Categories and Learning by Classification for Retrieval), a framework that shifts retrieval from traditional pair-based metric learning to a massive-scale classification paradigm.

The OCLEAR Framework

The OCLEAR framework rests on three primary pillars designed to improve data quality, fusion efficiency, and training scalability.

Category-Oriented Data Governance

Industrial data is often noisy and skewed toward popular items. OCLEAR employs a data governance strategy that organizes billions of images into millions of fine-grained categories (unique product IDs). By merging data from three sources—inherent product metadata, user click behavior, and unsupervised visual clustering—the model becomes robust to the inherent randomness and errors in human behavior.

Data governance strategy merging multiple signals to create robust product IDs.

Concept-Aware Modality Fusion (CAMF)

A core innovation of OCLEAR is the Concept-Aware Modality Fusion module. In IMMR, images provide specific visual details while text offers high-level abstract concepts. Standard fusion methods often fail to balance these, leading to abstract text "washing out" specific visual details.

OCLEAR uses a dual-tower architecture where the document side extracts textual "concepts" using external attention. These concepts then act as the Query (Q) in a cross-attention mechanism, where image features serve as the Key (K) and Value (V). This ensures the image remains the primary signal while the text provides complementary context.

The OCLEAR architecture featuring the dual-tower encoder and the Concept-Aware Modality Fusion (CAMF) module.

Scaling via Hybrid Parallel Training

Training on tens of millions of categories requires significant computational resources. OCLEAR utilizes a hybrid parallel training approach, splitting the classification layer (ID center proxy vectors) across multiple GPUs. To minimize the massive communication overhead between these units, the framework implements KNN-softmax, which calculates gradients only for the top-K most similar categories for any given sample, significantly accelerating convergence.

Experimental Results

OCLEAR was evaluated on the AliProduct dataset (12 million products, 1.1 billion images) and the Taobao Live (WAB) dataset.

  • State-of-the-Art Performance: OCLEAR achieved 78.41% Identical@1 accuracy on AliProduct, significantly outperforming established models like CLIP (71.68%) and Oscar (69.03%).
  • Ablation Success: Tests confirmed that the multi-modal fusion module provides a significant boost in Mean Average Precision (MAP) compared to image-only baselines.
  • Industrial Impact: Following deployment on Taobao’s image search system, an A/B test showed a +6.45% increase in actual purchases, proving that superior representation learning translates directly to business growth.

Conclusion

OCLEAR demonstrates that shifting from triplet-based retrieval to a massive-scale classification paradigm, combined with concept-aware multi-modal fusion, can significantly improve retrieval performance in large-scale industrial applications. Future work will focus on automated methods to handle long-tailed data distributions during the training process itself.

Summary updated

Abstract

The rise of multi-modal search requests from users has highlighted the importance of multi-modal retrieval (i.e. image-to-text or text-to-image retrieval), yet the more complex task of image-to-multi-modal retrieval, crucial for many industry applications, remains under-explored. To address this gap and promote further research, we introduce and define the concept of Image-to-Multi-Modal Retrieval (IMMR), a process designed to retrieve rich multi-modal (i.e. image and text) documents based on image queries. We focus on representation learning for IMMR and analyze three key challenges for it: 1) skewed data and noisy label in real-world industrial data, 2) the information-inequality between image and text modality of documents when learning representations, 3) effective and efficient training in large-scale industrial contexts. To tackle the above challenges, we propose a novel framework named organizing categories and learning by classification for retrieval (OCLEAR). It consists of three components: 1) a novel category-oriented data governance scheme coupled with a large-scale classification-based learning paradigm, which handles the skewed and noisy data from a data perspective. 2) model architecture specially designed for multi-modal learning, where information-inequality between image and text modality of documents is considered for modality fusion. 3) a hybrid parallel training approach for tackling large-scale training in industrial scenario. The proposed framework achieves SOTA performance on public datasets and has been deployed in a real-world industrial e-commence system, leading to significant business growth. Code will be made publicly available.
BibTeX
@misc{cheng2023mixer,
  author = {Zida Cheng and Shuai Xiao and Zhonghua Zhai and Xiaoyi Zeng and Weilin Huang},
  title = {{Mixer: Image to Multi-Modal Retrieval Learning for Industrial Application}},
  year = {2023},
  eprint = {2305.03972},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.03972},
}
Figure