Embedding Convolutions for Short Text Extreme Classification with Millions of Labels

Siddhant Kharbanda, Atmadeep Banerjee, Akash Palrecha, Rohit Babbar

Introduction

Extreme Multi-label Classification (XML) involves selecting the most relevant labels for an input from a pool of millions. This task is particularly challenging when dealing with short-text inputs, such as search queries or product titles, which typically consist of only 3 to 8 words. Unlike long-form documents, short queries are sparse, noisy, and often lack standard grammatical structure.

Current state-of-the-art solutions generally fall into two categories: massive transformer-based models, which are accurate but computationally prohibitive for real-time applications, and "frugal" models like Astec, which are faster but often suffer from training inefficiencies like "stale" negative samples. This paper introduces InceptionXML, a framework designed to achieve transformer-level accuracy with significantly lower computational overhead.

Methods: InceptionXML and SyncXML

The proposed approach consists of two primary components: a novel architecture called InceptionXML and a synchronized training pipeline named SyncXML.

InceptionXML Architecture

The InceptionXML encoder rethinks traditional text convolutions. Instead of convolving over the sequence of words, it performs embedding convolutions across the embedding dimensions themselves. This design is based on the hypothesis that different dimensions in a word vector represent coupled semantics (e.g., tense or sentiment), and capturing these patterns is more effective for short, word-order-independent queries.

To further enhance robustness, an Embedding Enhancement Module utilizes dual orthogonal attention layers. This allows information to flow across semantic dimensions before the convolutional layers, making the model less sensitive to variations in word order.

The InceptionXML architecture featuring the Embedding Enhancement Module and the SyncXML training pipeline.

SyncXML Training Pipeline

Scaling to millions of labels necessitates negative sampling. Most existing models use a separate "shortlister" to identify hard negative examples, but if this shortlister is not updated frequently, the main classifier learns from outdated data. SyncXML addresses this by training a meta-classifier and the primary classifier in lockstep. Additionally, "detached training" is employed during the initial 25% of the training process, allowing the meta-classifier to "warm up" and provide high-quality negative samples before fine-tuning the encoder.

Key Results

InceptionXML was evaluated on four standard short-text XML benchmarks, including AmazonTitles-670K and AmazonTitles-3M. The model outperformed existing state-of-the-art approaches (such as Astec and LightXML) on nearly all metrics across these datasets.

The most significant advantage of InceptionXML lies in its efficiency. It requires only 2% of the Giga-FLOPs used by transformer-based models and is an order of magnitude smaller in memory footprint. For instance, on the Amazon-670K dataset, InceptionXML occupies 1.4 GB compared to Astec's 10.9 GB.

Comparison of P@1 accuracy vs. training time and model size across various XML frameworks.

Conclusion

InceptionXML demonstrates that specialized architectures for short-text XML can outperform massive general-purpose models. By focusing on embedding-dimension convolutions and synchronized negative sampling, the framework achieves a "sweet spot" of high accuracy and extreme efficiency. Future work includes exploring the application of embedding convolutions to long-form text and deeper integration of label semantics.

Summary updated

BibTeX
@inproceedings{kharbanda2021embedding,
  author = {Siddhant Kharbanda and Atmadeep Banerjee and Akash Palrecha and Rohit Babbar},
  title = {{Embedding Convolutions for Short Text Extreme Classification with Millions of Labels}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2021},
}
Figure