Multi-modal Extreme Classification

Anshul Mittal, Kunal Dahiya, Shreya Malani, Janani Ramaswamy, Seba Ann Kuruvilla, Jitendra Ajmera, Keng-hao Chang, Sumeet Agarwal, Purushottam Kar, Manik Varma

Introduction

Extreme Multi-label Classification (XC) involves tagging data points with the most relevant subset of labels from a massive pool, often containing millions of possibilities. This technology powers critical applications like product recommendation on Amazon or query prediction in search engines. Traditionally, XC has relied heavily on textual data. While multi-modal models like CLIP have advanced joint representations of images and text, they are primarily designed for retrieval via embedding distances. They often lack the discriminative power of high-capacity, one-vs-all classifiers used in XC. Conversely, existing XC models struggle to integrate visual data efficiently at scale.

The MUFIN Framework

To bridge this gap, researchers introduced MUFIN (MUltimodal extreme classiFIcatioN). MUFIN is designed to be expressive enough for millions of classes while remaining efficient enough for millisecond-level inference. The architecture centers on four key components:

  1. Bag-of-Embeddings Representation: Instead of a single vector, each data point and label is represented as a "bag" of descriptors, incorporating both titles and multiple images.
  2. Self-Attention ($A_S$): This block enables interaction between different modalities (text and images) within a single product, creating a cohesive multi-modal representation.
  3. Cross-Attention ($A_C$): This mechanism allows the model to adapt a product's representation to a specific label during comparison. By focusing on relevant visual or textual features, it enables fine-grained matching.
  4. Modular Training Pipeline: To manage the $O(NL)$ complexity of millions of labels, MUFIN employs a four-stage process: Siamese pre-training, augmented retrieval for "hard negatives," and fine-tuning of cross-attention and one-vs-all classifiers.

Experimental Setup

MUFIN was evaluated on three diverse benchmarks:

  • MM-AmazonTitles-300K: A newly curated dataset of 300,000 products, each with a title and multiple images.
  • A2Q-4M: A massive dataset for bid-query prediction with 4 million labels, derived from Bing search logs.
  • Polyvore-Disjoint: A fashion dataset used to evaluate zero-shot capabilities on labels never seen during training.

Key Insights and Results

A core finding is the superiority of dedicated classifier layers over pure embedding-based retrieval. Adding a classifier layer ($w_l$) on top of multi-modal embeddings significantly boosts accuracy compared to models like CLIP.

A successful multi-modal match for a motorcycle-shaped clock.

The cross-attention mechanism is particularly effective for "fine-grained" matching. For instance, when processing a query for a "motorcycle-shaped clock," the model can distinguish between relevant accessories and visually similar but semantically different items, such as motorcycle parts.

Performance Metrics: * Accuracy: On the Amazon dataset, MUFIN achieved a 52.3% Precision@1, outperforming the text-only SiameseXML by ~3.5% and multi-modal baselines by up to 12%. * Scalability: By utilizing Approximate Nearest Neighbor Search (ANNS) for label shortlisting, MUFIN maintains inference times of 3-4 milliseconds. * Zero-Shot Performance: On the Polyvore dataset, MUFIN surpassed the previous state-of-the-art by 3-4%, demonstrating robust generalization.

Conclusion and Future Work

MUFIN demonstrates that multi-modal intelligence can be scaled to millions of labels without sacrificing efficiency. However, challenges remain. There is a persistent trade-off between performance on "head" (popular) versus "tail" (rare) labels. Future work will explore integrating richer metadata, such as taxonomies and pricing, and reducing the dependency on the initial retrieval shortlist quality.

Summary updated

Abstract

This paper develops the MUFIN technique for extreme classification (XC) tasks with millions of labels where data points and labels are endowed with visual and textual descriptors. Applications of MUFIN to product-to-product recommendation and bid query prediction over several millions of products are presented. Contemporary multi-modal methods frequently rely on purely embedding-based methods. On the other hand, XC methods utilize classifier architectures to offer superior accuracies than embedding only methods but mostly focus on text-based categorization tasks. MUFIN bridges this gap by reformulating multimodal categorization as an XC problem with several millions of labels. This presents the twin challenges of developing multi-modal architectures that can offer embeddings sufficiently expressive to allow accurate categorization over millions of labels; and training and inference routines that scale logarithmically in the number of labels. MUFIN develops an architecture based on cross-modal attention and trains it in a modular fashion using pre-training and positive and negative mining. A novel product-to-product recommendation dataset MM-AmazonTitles-300K containing over 300K products was curated from publicly available amazon.com listings with each product endowed with a title and multiple images. On the MM- AmazonTitles-300K and Polyvore datasets, and a dataset with over 4 million labels curated from click logs of the Bing search engine, MUFIN offered at least 3% higher accuracy than leading text-based, image-based and multi-modal techniques.
BibTeX
@inproceedings{mittal2022multi,
  author = {Anshul Mittal and Kunal Dahiya and Shreya Malani and Janani Ramaswamy and Seba Ann Kuruvilla and Jitendra Ajmera and Keng-hao Chang and Sumeet Agarwal and Purushottam Kar and Manik Varma},
  title = {{Multi-modal Extreme Classification}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
}
Figure