GUDN: A novel guide network with label reinforcement strategy for extreme multi-label text classification

Qing Wang, Jia Zhu, Hongji Shu, Kwame Omono Asamoah, Jianyang Shi, Cong ZhouJ. King Saud Univ. Comput. Inf. Sci 2022

Abstract

In natural language processing, extreme multi-label text classification (XMTC) is an essential task involving the retrieval of relevant labels from extremely large sets. While large-scale pre-trained models have advanced this field, the semantic gap between long documents and short label descriptions remains a significant challenge. This paper introduces a new guide network (GUDN) designed to fine-tune pre-trained models by effectively exploring the latent space between texts and labels. GUDN utilizes raw label semantics and a novel label reinforcement strategy to narrow this semantic gap. Experimental results demonstrate that GUDN achieves state-of-the-art performance on the Eurlex-4K dataset and competitive results on other major benchmarks.

Introduction

Extreme Multi-label Text Classification (XMTC) involves assigning the most relevant tags to a document from a pool that can contain hundreds of thousands of options, such as categorizing products on Amazon or tagging Wikipedia articles. A major bottleneck in current methods is the semantic gap. Traditional approaches often treat labels as simple numerical indices, ignoring the inherent meaning of the label words. Furthermore, a "length dichotomy" exists: documents are often thousands of words long, while labels consist of only a few words. GUDN (Guide Network) addresses these issues by treating labels as semantic descriptions and aligning them with text features in a shared latent space.

Methodology

The GUDN architecture is built upon a BERT backbone and consists of three primary components designed to bridge the gap between text and label representations.

1. Feature Extractor

Unlike standard models that only embed the input text, GUDN uses BERT to extract features for both the input document and the raw text of the labels. By feeding the actual words of the labels into the model, GUDN leverages the semantic knowledge already present in pre-trained models.

2. The Guide Network

The Guide Network is the core innovation, using two "guides" to align features. The Feature Loss (Mean Squared Error) forces text features to be closer to label features in the latent space. Simultaneously, the Link Loss establishes a direct mapping between label features and ground-truth labels.

3. Label Reinforcement Strategy

To resolve the length mismatch between documents and labels, GUDN "reinforces" the label input. Since BERT expects 512 tokens, the model fills the label input window by either repeating the label sequence (Ordered Fill) or randomly shuffling and repeating them (Disordered Fill) until the limit is reached. Interestingly, the "disordered" fill method often outperforms the ordered one by preventing the Transformer from learning specific sequence biases.

The GUDN architecture overview, highlighting the Label Reinforcement Strategy and the Guide Network components.

Experimental Results

The authors evaluated GUDN on four benchmark XMTC datasets: Eurlex-4K, AmazonCat-13K, Wiki10-31K, and Wiki-500K. The model was trained using a joint loss function: $L_{overall} = L_{guide} + L_{class}$.

Performance Metrics

GUDN achieved state-of-the-art (SOTA) results on the Eurlex-4K dataset, outperforming strong competitors like LightXML and X-Transformer across all Precision metrics, including a P@1 of 88.13%. On the massive Wiki-500K dataset, which contains half a million labels, GUDN remained highly competitive, proving the scalability of the guide network strategy.

Precision metrics (P@k) for GUDN compared to existing SOTA models across benchmark datasets.

Key Insights

  • Sensitivity to Semantics: GUDN performs best on datasets with "clean" natural language labels (like Eurlex). Its performance dips when labels are symbolic or nonsensical (e.g., "!!! albums"), confirming that the model's strength lies in processing human-readable meaning.
  • Ablation Studies: Removing the Guide Network led to a significant drop in accuracy, confirming that label-semantic guidance is a critical driver of performance.

Conclusion and Future Work

GUDN demonstrates that treating labels as semantic data rather than indices significantly improves XMTC performance. While highly effective for natural language labels, future work will focus on handling "symbolic" labels that lack clear semantic meaning. Additionally, while experiments with the Longformer (supporting 4096 tokens) showed mixed results due to added noise, further investigation into input length optimization remains a priority for improving classification in long-form documents.

Summary updated

Abstract

In natural language processing, extreme multi-label text classification is an emerging but essential task. The problem of extreme multi-label text classification (XMTC) is to recall some of the most relevant labels for a text from an extremely large label set. Large-scale pre-trained models have brought a new trend to this problem. Though the large-scale pre-trained models have made significant achievements on this problem, the valuable fine-tuned methods have yet to be studied. Though label semantics have been introduced in XMTC, the vast semantic gap between texts and labels has yet to gain enough attention. This paper builds a new guide network (GUDN) to help fine-tune the pre-trained model to instruct classification later. Furthermore, GUDN uses raw label semantics combined with a helpful label reinforcement strategy to effectively explore the latent space between texts and labels, narrowing the semantic gap, which can further improve predicted accuracy. Experimental results demonstrate that GUDN outperforms state-of-the-art methods on Eurlex-4k and has competitive results on other popular datasets. In an additional experiment, we investigated the input lengths' influence on the Transformer-based model's accuracy. Our source code is released at https://t.hk.uy/aFSH.
BibTeX
@article{wang2022gudn,
  author = {Qing Wang and Jia Zhu and Hongji Shu and Kwame Omono Asamoah and Jianyang Shi and Cong Zhou},
  title = {{GUDN: A novel guide network with label reinforcement strategy for extreme multi-label text classification}},
  journal = {J. King Saud Univ. Comput. Inf. Sci.},
  year = {2022},
}
Figure