Rank4Class: A Ranking Formulation for Multiclass Classification

Nan Wang, Zhen Qin, Le Yan, Honglei Zhuang, Xuanhui Wang, Michael Bendersky, Marc NajorkarXiv 2021 · arXiv

In the field of deep learning, multiclass classification (MCC) is a fundamental task. Whether labeling an image or categorizing a news article, the standard approach typically involves feeding an input into a powerful encoder (e.g., BERT or ResNet), applying a dense layer, and training with Softmax Cross Entropy. However, a recent study by researchers at Google and the University of Virginia, titled Rank4Class, suggests that reframing classification as a Learning-to-Rank (LTR) problem can significantly enhance model performance.

Reframing Classification as Ranking

Rank4Class reformulates MCC as a ranking task where the objective is to place the correct class at the top of a ranked list. By adopting ranking-specific loss functions and "interaction layers" common in Information Retrieval, the authors demonstrate consistent performance improvements across major text and image benchmarks without altering the underlying backbone encoders.

Comparison of standard Multiclass Classification, its Ranking View, and the proposed Ranking Generalization (Rank4Class).

Addressing Architectural and Metric Limitations

The authors identify two primary shortcomings in traditional MCC:

  1. Metric Limitations: Standard evaluation often relies on Top-K Accuracy, a "hard cut" metric. If a correct class moves from rank 10 to rank 6, Top-5 Accuracy remains unchanged. The authors advocate for NDCG (Normalized Discounted Cumulative Gain), which provides a more granular view of model performance by rewarding higher rankings even below the top-1 spot.
  2. Architectural Rigidity: The standard "dot product" between an instance embedding and a class weight is a relatively simple matching mechanism. Rank4Class introduces more complex Interaction Layers to capture non-linear relationships between inputs and classes.

Methodology

The core concept is to treat the input instance as a query and the potential classes as items to be ranked.

  • Ranking Losses: Beyond Softmax Cross Entropy (SoftmaxCE), the framework utilizes LTR losses such as PairLogLoss (optimizing pairs of classes) and ApproxNDCG (directly optimizing for the NDCG metric).
  • Enhanced Interaction Layers: Instead of a simple linear projection, the authors tested advanced interaction methods:
  • LC+MLP: A "Latent Cross" (element-wise multiplication) followed by a Multi-Layer Perceptron.
  • Concat+MLP: Concatenating instance and class embeddings before passing them through an MLP.

Experimental Results

The framework was validated across diverse datasets:

  • Text: GoEmotions (fine-grained emotions) and MIND (news categories) using BERT and ELECTRA backbones.
  • Image: ImageNet and CIFAR-10 using ResNet50 and VGG16 backbones.

Key findings include:

  • Significant Error Reduction: On the GoEmotions dataset with ELECTRA, the authors observed a 17.22% relative reduction in Top-5 Error. On CIFAR-10 with VGG16, they achieved a 25% relative reduction.
  • Sensitivity of NDCG: Experiments showed that ranking metrics are more sensitive than Top-K accuracy, successfully distinguishing between models that appeared identical under standard metrics.
  • Theoretical Grounding: The paper proves that SoftmaxCE is a bound on ranking metrics like Mean Reciprocal Rank, explaining its historical success while positioning it as a specific case of a broader ranking framework.

Conclusion and Future Directions

Rank4Class demonstrates that the ranking perspective is a general-purpose enhancement for classification tasks. While ranking losses may require more precise hyperparameter tuning, the gains in accuracy and model sensitivity are substantial. Future research may explore using class metadata (like names or descriptions) to create richer embeddings and scaling the framework to "extreme" classification problems involving millions of labels.

Summary updated

Abstract

Multiclass classification (MCC) is a fundamental machine learning problem of classifying each instance into one of a predefined set of classes. In the deep learning era, extensive efforts have been spent on developing more powerful neural embedding models to better represent the instance for improving MCC performance. In this paper, we do not aim to propose new neural models for instance representation learning, but to show that it is promising to boost MCC performance with a novel formulation through the lens of ranking. In particular, by viewing MCC as to rank classes for an instance, we first argue that ranking metrics, such as Normalized Discounted Cumulative Gain, can be more informative than the commonly used Top-$K$ metrics. We further demonstrate that the dominant neural MCC recipe can be transformed to a neural ranking framework. Based on such generalization, we show that it is intuitive to leverage advanced techniques from the learning to rank literature to improve the MCC performance out of the box. Extensive empirical results on both text and image classification tasks with diverse datasets and backbone neural models show the value of our proposed framework.
BibTeX
@misc{wang2021rank,
  author = {Nan Wang and Zhen Qin and Le Yan and Honglei Zhuang and Xuanhui Wang and Michael Bendersky and Marc Najork},
  title = {{Rank4Class: A Ranking Formulation for Multiclass Classification}},
  year = {2021},
  eprint = {2112.09727},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2112.09727},
}
Figure