Introduction
Multi-label text classification (MLTC) involves assigning multiple relevant tags to a single document. While foundational to many NLP applications, MLTC faces significant challenges in real-world scenarios where label distributions are heavily skewed. This paper, presented at EMNLP 2021, addresses the persistent issue of classifying text when some labels appear thousands of times while others appear only a handful, particularly when rare labels are "linked" to common ones.
The Challenges: Long Tails and Label Linkage
Standard models often struggle with two major hurdles in MLTC:
- Long-Tailed Distribution: Most datasets follow a power-law distribution where a few "head" labels are frequent, and the vast majority of "tail" labels are extremely rare. Standard models typically optimize for overall error, leading them to ignore these rare classes.
- Label Linkage (Co-occurrence): Rare labels often co-occur with common ones. For instance, a rare label like "Nickel" might frequently appear alongside the common label "Acquisitions." Traditional resampling methods that attempt to boost rare labels inadvertently oversample the common labels they are linked to, reinforcing existing biases.
As shown in the figure above, the Reuters-21578 dataset exhibits a sharp drop-off in article frequency across labels, paired with a complex web of co-occurrences. This "linkage" makes simple data-level balancing difficult because you cannot isolate a rare label from the common ones it is attached to.
Proposed Methodology: Distribution-Balanced Loss
To combat these issues, the authors adapt Distribution-Balanced (DB) loss—a technique originally developed for computer vision—to the NLP domain. DB loss modifies the learning process through two primary mechanisms:
- Rebalanced Weighting: This calculates weights based on label co-occurrence frequencies. By doing so, it prevents common labels from being over-emphasized when the model attempts to learn from samples containing rare labels.
- Negative-Tolerant Regularization (NTR): In imbalanced settings, rare classes are frequently treated as "negative" examples, leading to "over-suppression." NTR reduces the penalty for these classes, allowing the model to be more tolerant and effective at predicting them.
The study also evaluates other balancing functions like Focal Loss (FL), which focuses on "hard" examples, and Class-Balanced Loss (CB), which reweights based on the "effective number of samples."
Experimental Results
The researchers tested these methods on the Reuters-21578 news dataset and the massive PubMed (BioASQ) dataset, which contains over 18,000 labels.
The results demonstrate that DB loss is a superior alternative to standard Binary Cross Entropy (BCE):
- Reuters-21578: DB loss improved Macro-F1 from 47.32 (BCE) to 64.47. Crucially, for tail labels (appearing $\le$ 8 times), the Micro-F1 rose from 0.00 to 48.89.
- PubMed: On this extreme long-tail dataset, DB loss achieved a tail Micro-F1 of 24.19, whereas BCE failed to predict rare labels entirely (0.00 F1).
Importantly, DB loss achieved these gains without degrading performance on common "head" labels, proving its robustness across the entire distribution.
Conclusion
This work highlights that loss-function-based balancing is often more effective than traditional resampling in multi-label settings because it is architecture-agnostic and handles label linkage more precisely. The success of Distribution-Balanced loss suggests that cross-pollination between computer vision and NLP can provide powerful solutions for shared challenges like extreme class imbalance.
Summary updated