Balancing Methods for Multi-label Text Classification with Long-Tailed Class Distribution

Yi Huang, Buse Giledereli, Abdullatif Köksal, Arzucan Özgür, Elif Ozkirimli

Introduction

Multi-label text classification (MLTC) involves assigning multiple relevant tags to a single document. While foundational to many NLP applications, MLTC faces significant challenges in real-world scenarios where label distributions are heavily skewed. This paper, presented at EMNLP 2021, addresses the persistent issue of classifying text when some labels appear thousands of times while others appear only a handful, particularly when rare labels are "linked" to common ones.

The Challenges: Long Tails and Label Linkage

Standard models often struggle with two major hurdles in MLTC:

  1. Long-Tailed Distribution: Most datasets follow a power-law distribution where a few "head" labels are frequent, and the vast majority of "tail" labels are extremely rare. Standard models typically optimize for overall error, leading them to ignore these rare classes.
  2. Label Linkage (Co-occurrence): Rare labels often co-occur with common ones. For instance, a rare label like "Nickel" might frequently appear alongside the common label "Acquisitions." Traditional resampling methods that attempt to boost rare labels inadvertently oversample the common labels they are linked to, reinforcing existing biases.
Long-tail distribution and label co-occurrence heatmap for the Reuters-21578 dataset.

As shown in the figure above, the Reuters-21578 dataset exhibits a sharp drop-off in article frequency across labels, paired with a complex web of co-occurrences. This "linkage" makes simple data-level balancing difficult because you cannot isolate a rare label from the common ones it is attached to.

Examples of label linkage where rare labels (e.g., nickel) co-occur with more frequent ones (e.g., acq, copper).

Proposed Methodology: Distribution-Balanced Loss

To combat these issues, the authors adapt Distribution-Balanced (DB) loss—a technique originally developed for computer vision—to the NLP domain. DB loss modifies the learning process through two primary mechanisms:

  • Rebalanced Weighting: This calculates weights based on label co-occurrence frequencies. By doing so, it prevents common labels from being over-emphasized when the model attempts to learn from samples containing rare labels.
  • Negative-Tolerant Regularization (NTR): In imbalanced settings, rare classes are frequently treated as "negative" examples, leading to "over-suppression." NTR reduces the penalty for these classes, allowing the model to be more tolerant and effective at predicting them.

The study also evaluates other balancing functions like Focal Loss (FL), which focuses on "hard" examples, and Class-Balanced Loss (CB), which reweights based on the "effective number of samples."

Experimental Results

The researchers tested these methods on the Reuters-21578 news dataset and the massive PubMed (BioASQ) dataset, which contains over 18,000 labels.

The results demonstrate that DB loss is a superior alternative to standard Binary Cross Entropy (BCE):

  • Reuters-21578: DB loss improved Macro-F1 from 47.32 (BCE) to 64.47. Crucially, for tail labels (appearing $\le$ 8 times), the Micro-F1 rose from 0.00 to 48.89.
  • PubMed: On this extreme long-tail dataset, DB loss achieved a tail Micro-F1 of 24.19, whereas BCE failed to predict rare labels entirely (0.00 F1).

Importantly, DB loss achieved these gains without degrading performance on common "head" labels, proving its robustness across the entire distribution.

Conclusion

This work highlights that loss-function-based balancing is often more effective than traditional resampling in multi-label settings because it is architecture-agnostic and handles label linkage more precisely. The success of Distribution-Balanced loss suggests that cross-pollination between computer vision and NLP can provide powerful solutions for shared challenges like extreme class imbalance.

Summary updated

Abstract

Multi-label text classification is a challenging task because it requires capturing label dependencies. It becomes even more challenging when class distribution is long-tailed. Resampling and re-weighting are common approaches used for addressing the class imbalance problem, however, they are not effective when there is label dependency besides class imbalance because they result in oversampling of common labels. Here, we introduce the application of balancing loss functions for multi-label text classification. We perform experiments on a general domain dataset with 90 labels (Reuters-21578) and a domain-specific dataset from PubMed with 18211 labels. We find that a distribution-balanced loss function, which inherently addresses both the class imbalance and label linkage problems, outperforms commonly used loss functions. Distribution balancing methods have been successfully used in the image recognition field. Here, we show their effectiveness in natural language processing. Source code is available at https://github.com/Roche/BalancedLossNLP.
BibTeX
@inproceedings{huang2021balancing,
  author = {Yi Huang and Buse Giledereli and Abdullatif Köksal and Arzucan Özgür and Elif Ozkirimli},
  title = {{Balancing Methods for Multi-label Text Classification with Long-Tailed Class Distribution}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.643},
}
Figure