Light-weight Deep Extreme Multilabel Classification

Istasis Mishra, Arpan Dasgupta, Pratik Jawanpuria, Bamdev Mishra, Pawan Kumar

TL;DR

This paper introduces LightDXML, a novel deep learning framework for extreme multi-label classification that significantly reduces training time, model size, and prediction time while maintaining state-of-the-art accuracy. It achieves this by using label embeddings for negative sampling and a cyclic training procedure, eliminating the need for costly re-ranker modules.

What Problem Does This Paper Address?

Extreme multi-label (XML) classification deals with scenarios where a single data point can be assigned to one of millions of possible labels. This is common in applications like recommending advertiser bid phrases, tagging web pages, or predicting product attributes. The primary challenge in XML is scalability: as the number of labels grows exponentially, traditional classification methods become computationally infeasible. Existing deep learning approaches, while achieving high accuracy, often suffer from excessive training times, large memory footprints, and slow prediction speeds. Furthermore, many methods struggle with "tail labels" – those that appear infrequently in the dataset – leading to poor generalization for these rare labels.

Methods

LightDXML builds upon the DeepXML framework but introduces several key modifications to improve efficiency and performance. Instead of relying on feature embeddings for negative sampling, LightDXML utilizes label embeddings. These embeddings are learned in a low-dimensional space and are used to identify the "hardest" negative labels for training.

The core of LightDXML's innovation lies in its cyclic training procedure. This involves iteratively optimizing three components:

  1. Label Embeddings: Learning dense, low-dimensional representations for each label.
  2. Label Shortlisting: Identifying a small set of relevant negative labels for each data point using the learned label embeddings.
  3. Extreme Classifier Training: Training a one-vs-all classifier using the shortlisted labels and updated feature representations.

Crucially, LightDXML does not freeze the feature representations during training. This allows for continuous refinement and avoids the need for a separate re-ranker module, which is common in other methods like Astec (a previous DeepXML implementation) to compensate for frozen features. By cyclically updating label embeddings and feature representations, LightDXML ensures that the negative samples remain relevant throughout training, leading to better accuracy and eliminating the overhead of a re-ranker.

Datasets

The paper evaluates LightDXML on five publicly available multi-label datasets:

  • EURLex-4K
  • AmazonCat-13K
  • Wiki10-31K
  • WikiSee Also Titles-350K
  • AmazonTitles-670K

These datasets vary in size and number of labels, with the latter two being particularly large, allowing the authors to demonstrate scalability.

Key Insights and Ideas

  • Label Embeddings are Key for Efficiency: Using label embeddings for negative sampling, rather than feature embeddings, significantly streamlines the process and allows for more effective identification of "hard" negatives.
  • Cyclic Training Solves the "Frozen Feature" Problem: By iteratively updating label embeddings and feature representations, LightDXML avoids the need to freeze features. This continuous refinement leads to better accuracy and eliminates the necessity of a re-ranker module.
  • No Re-ranker Needed: The cyclic training approach inherently addresses the issues that typically necessitate a re-ranker, leading to substantial savings in training time, prediction time, and model size.
  • Tail Label Performance Matters: The paper emphasizes the importance of tail label prediction accuracy (measured by PSP@k) and shows LightDXML performs competitively or better than other methods on this metric.

Key Results

LightDXML demonstrates significant improvements across several metrics:

  • Speed: Achieves 2-3x speedup in training time and prediction time compared to the state-of-the-art Astec.
  • Memory: Reduces model size by 2-3x compared to Astec.
  • Accuracy: Maintains state-of-the-art precision scores (P@k) and often achieves the best propensity scores (PSP@k), indicating strong performance on tail labels.
  • Efficiency: On larger datasets, LightDXML requires significantly less memory (e.g., 7x less for AmazonCat-13K compared to Astec) while retaining competitive accuracy.

Limitations and Future Work

  • Hyperparameter Tuning: The paper acknowledges that hyperparameter tuning (e.g., β, k, emodel, elabel, êlabel) is crucial for optimal performance, and these values are listed for the tested datasets.
  • Comparison Scope: While the paper compares against many state-of-the-art methods, it notes that some methods are excluded due to not reporting train time or having excessively large model sizes.
  • Future Work: The authors suggest that the cyclic update procedure is a promising direction for future research in extreme multi-label classification. They also implicitly suggest that further exploration of label embedding techniques and their integration into deep learning frameworks could yield additional benefits.

Summary updated

BibTeX
@inproceedings{mishra2023light,
  author = {Istasis Mishra and Arpan Dasgupta and Pratik Jawanpuria and Bamdev Mishra and Pawan Kumar},
  title = {{Light-weight Deep Extreme Multilabel Classification}},
  booktitle = {IJCNN},
  year = {2023},
  doi = {10.1109/ijcnn54540.2023.10191716},
}

Loading related content...

Figure