Label-Specific Feature Augmentation for Long-Tailed Multi-Label Text Classification

Pengyu Xu, Lin Xiao, Bing Liu, Sijin Lu, Liping Jing, Jian YuAAAI 2023 · OpenReview

Abstract

Multi-label text classification (MLTC) involves tagging a document with its most relevant subset of labels from a label set. In real applications, labels usually follow a long-tailed distribution, where most labels (called as tail-label) only contain a small number of documents and limit the performance of MLTC. To facilitate this low-resource problem, researchers introduced a simple but effective strategy, data augmentation (DA). However, most existing DA approaches struggle in multi-label settings. The main reason is that the augmented documents for one label may inevitably influence the other co-occurring labels and further exaggerate the long-tailed problem. To mitigate this issue, we propose a new pair-level augmentation framework for MLTC, called Label-Specific Feature Augmentation (LSFA), which merely augments positive feature-label pairs for the tail-labels. LSFA contains two main parts. The first is for label-specific document representation learning in the high-level latent space, the second is for augmenting tail-label features in latent space by transferring the documents second-order statistics (intra-class semantic variations) from head labels to tail labels. At last, we design a new loss function for adjusting classifiers based on augmented datasets. The whole learning procedure can be effectively trained. Comprehensive experiments on benchmark datasets have shown that the proposed LSFA outperforms the state-of-the-art counterparts.

Introduction

In Multi-Label Text Classification (MLTC), a single document is often associated with multiple tags. A significant challenge in this field is the long-tailed distribution of labels: a small number of "head" labels appear frequently, while the vast majority of "tail" labels have very few examples.

Standard Data Augmentation (DA) techniques, such as word swapping or back-translation, often fail in MLTC because labels co-occur. Augmenting a document to bolster a rare label can inadvertently distort the patterns of co-occurring common labels. This "sample-level" approach lacks the precision required for multi-label environments. To address this, the Label-Specific Feature Augmentation (LSFA) framework shifts augmentation from the text level to the latent feature space, focusing on specific feature-label pairs.

Methodology

The LSFA framework consists of two main stages designed to decouple representations and transfer semantic knowledge from head to tail labels.

The LSFA architecture, featuring Decoupled Representation Learning and Head-to-Tail Feature Augmentation.

1. Decoupled Representation Learning

Instead of a single global representation, LSFA employs a label-specific encoder (using BiLSTM and Attention) to extract unique features for each label. To organize the latent space, a Prototypical Supervised Contrastive (PSC) loss is introduced. This loss function pulls features of the same label toward a central "prototype" while pushing different label prototypes apart, ensuring a clean and separable feature space.

2. Head-to-Tail Feature Augmentation

Using a Prototype-based Variational Autoencoder (PVAE), the model learns "intra-class semantic variations"—the range of ways a feature can change while maintaining its label identity—from data-rich head labels. This learned variance is then transferred to tail labels. By applying these variations to a tail label's prototype, the model generates synthetic positive feature-label pairs, effectively augmenting the training set without the side effects of document-level manipulation.

Experiments and Results

LSFA was evaluated on three benchmark datasets: AAPD (academic abstracts), RCV1 (newswire), and EUR-Lex (legal documents). EUR-Lex is particularly challenging, with 97% of its labels categorized as rare.

Comparison of P@5 performance between LSFA and HTTN across label bins sorted by frequency.

The experimental results demonstrate that LSFA consistently outperforms state-of-the-art models like HTTN and LDGN. On the EUR-Lex dataset, LSFA achieved a P@1 of 83.75%, a significant improvement over the previous SOTA of 81.03%. Most notably, the model showed superior performance on tail labels, improving the PSP@5 metric (Propensity-Scored Precision) by 10.82% on EUR-Lex. As shown in the performance breakdown across label frequency bins, LSFA maintains a clear advantage even as label frequency decreases, confirming its effectiveness in addressing the long-tail problem.

Conclusion

LSFA introduces a novel approach to multi-label data augmentation by operating at the pair-level in the latent feature space. By decoupling representations and transferring semantic variance from head to tail labels, it overcomes the limitations of traditional sample-level augmentation. Future work may explore integrating these feature-level improvements with advanced input-level text augmentation and scaling the architecture for extreme multi-label tasks.

Summary updated

Abstract

Multi-label text classification (MLTC) involves tagging a document with its most relevant subset of labels from a label set. In real applications, labels usually follow a long-tailed distribution, where most labels (called as tail-label) only contain a small number of documents and limit the performance of MLTC. To facilitate this low-resource problem, researchers introduced a simple but effective strategy, data augmentation (DA). However, most existing DA approaches struggle in multi-label settings. The main reason is that the augmented documents for one label may inevitably influence the other co-occurring labels and further exaggerate the long-tailed problem. To mitigate this issue, we propose a new pair-level augmentation framework for MLTC, called Label-Specific Feature Augmentation (LSFA), which merely augments positive feature-label pairs for the tail-labels. LSFA contains two main parts. The first is for label-specific document representation learning in the high-level latent space, the second is for augmenting tail-label features in latent space by transferring the documents second-order statistics (intra-class semantic variations) from head labels to tail labels. At last, we design a new loss function for adjusting classifiers based on augmented datasets. The whole learning procedure can be effectively trained. Comprehensive experiments on benchmark datasets have shown that the proposed LSFA outperforms the state-of-the-art counterparts.
BibTeX
@inproceedings{xu2023label,
  author = {Pengyu Xu and Lin Xiao and Bing Liu and Sijin Lu and Liping Jing and Jian Yu},
  title = {{Label-Specific Feature Augmentation for Long-Tailed Multi-Label Text Classification}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  url = {https://openreview.net/forum?id=ypno4CM4T0},
}
Figure