Abstract
In natural language processing, extreme multi-label text classification (XMTC) is an essential task involving the retrieval of relevant labels from extremely large sets. While large-scale pre-trained models have advanced this field, the semantic gap between long documents and short label descriptions remains a significant challenge. This paper introduces a new guide network (GUDN) designed to fine-tune pre-trained models by effectively exploring the latent space between texts and labels. GUDN utilizes raw label semantics and a novel label reinforcement strategy to narrow this semantic gap. Experimental results demonstrate that GUDN achieves state-of-the-art performance on the Eurlex-4K dataset and competitive results on other major benchmarks.
Introduction
Extreme Multi-label Text Classification (XMTC) involves assigning the most relevant tags to a document from a pool that can contain hundreds of thousands of options, such as categorizing products on Amazon or tagging Wikipedia articles. A major bottleneck in current methods is the semantic gap. Traditional approaches often treat labels as simple numerical indices, ignoring the inherent meaning of the label words. Furthermore, a "length dichotomy" exists: documents are often thousands of words long, while labels consist of only a few words. GUDN (Guide Network) addresses these issues by treating labels as semantic descriptions and aligning them with text features in a shared latent space.
Methodology
The GUDN architecture is built upon a BERT backbone and consists of three primary components designed to bridge the gap between text and label representations.
1. Feature Extractor
Unlike standard models that only embed the input text, GUDN uses BERT to extract features for both the input document and the raw text of the labels. By feeding the actual words of the labels into the model, GUDN leverages the semantic knowledge already present in pre-trained models.
2. The Guide Network
The Guide Network is the core innovation, using two "guides" to align features. The Feature Loss (Mean Squared Error) forces text features to be closer to label features in the latent space. Simultaneously, the Link Loss establishes a direct mapping between label features and ground-truth labels.
3. Label Reinforcement Strategy
To resolve the length mismatch between documents and labels, GUDN "reinforces" the label input. Since BERT expects 512 tokens, the model fills the label input window by either repeating the label sequence (Ordered Fill) or randomly shuffling and repeating them (Disordered Fill) until the limit is reached. Interestingly, the "disordered" fill method often outperforms the ordered one by preventing the Transformer from learning specific sequence biases.
Experimental Results
The authors evaluated GUDN on four benchmark XMTC datasets: Eurlex-4K, AmazonCat-13K, Wiki10-31K, and Wiki-500K. The model was trained using a joint loss function: $L_{overall} = L_{guide} + L_{class}$.
Performance Metrics
GUDN achieved state-of-the-art (SOTA) results on the Eurlex-4K dataset, outperforming strong competitors like LightXML and X-Transformer across all Precision metrics, including a P@1 of 88.13%. On the massive Wiki-500K dataset, which contains half a million labels, GUDN remained highly competitive, proving the scalability of the guide network strategy.
Key Insights
- Sensitivity to Semantics: GUDN performs best on datasets with "clean" natural language labels (like Eurlex). Its performance dips when labels are symbolic or nonsensical (e.g., "!!! albums"), confirming that the model's strength lies in processing human-readable meaning.
- Ablation Studies: Removing the Guide Network led to a significant drop in accuracy, confirming that label-semantic guidance is a critical driver of performance.
Conclusion and Future Work
GUDN demonstrates that treating labels as semantic data rather than indices significantly improves XMTC performance. While highly effective for natural language labels, future work will focus on handling "symbolic" labels that lack clear semantic meaning. Additionally, while experiments with the Longformer (supporting 4096 tokens) showed mixed results due to added noise, further investigation into input length optimization remains a priority for improving classification in long-form documents.
Summary updated