Abstract
Assigning a subset of labels from a fixed pool to a given input text is a fundamental classification problem with diverse applications, including recommender systems. Currently, two separate research streams address this: Hierarchical Text Classification (HTC), which focuses on smaller label pools (hundreds) with semantic hierarchies, and eXtreme Multi-Label Text Classification (XML), which handles massive label pools (up to millions) without explicit hierarchies. This paper investigates the bidirectional transferability of state-of-the-art models between these domains. We find that while XML models like CascadeXML and XR-Transformer excel on HTC tasks—often outperforming specialized HTC models—HTC models like HBGL and HGLCR struggle to scale to XML datasets. Our results suggest that the artificial hierarchies constructed by XML models are highly effective even when semantic taxonomies are available.
Introduction
In Natural Language Processing (NLP), multi-label classification is typically divided into two silos. HTC datasets feature human-curated taxonomies (e.g., library classifications), while XML datasets deal with such extreme scale that models must build their own internal "artificial" trees for computational efficiency. Because these fields use different evaluation metrics (F1-scores for HTC vs. Precision@k for XML), they rarely interact. This study breaks down these barriers to ask whether XML models can solve HTC problems and if HTC models can scale to the "extreme" level.
Methodology
To facilitate a fair comparison, the researchers established a bidirectional transfer framework:
- Metric Harmonization: The authors introduced R-Precision as a primary metric. Unlike Precision@k or F1, R-Precision evaluates the top $r$ predictions, where $r$ is the actual number of ground-truth labels for a document, providing a threshold-independent view of ranking quality.
- Dataset Conversion:
- XML on HTC: The semantic hierarchy was "collapsed," treating labels as a flat set and allowing XML models to build internal clusters.
- HTC on XML: Synthetic hierarchies were generated for XML datasets using Positive Instance Feature Aggregation (PIFA) combined with balanced k-means.
- Model Selection: The study compared transformer-based models: HGCLR and HBGL (HTC domain) against XR-Transformer and CascadeXML (XML domain).
Datasets
The study utilized six benchmark datasets spanning both domains. The primary distinction lies in the label count and the presence of a predefined hierarchy.
Key Insights
The Scalability Wall
A significant finding was the inability of HTC models to handle extreme scales. Architectures like HGCLR and HBGL often rely on $O(n^2)$ components, such as dense attention or complex graph encoders. These models frequently crashed or exceeded memory limits when faced with the 670,000 labels of the Amazon-670K dataset.
Data-Driven vs. Semantic Hierarchies
Surprisingly, XML-style artificial hierarchies—built on label frequency and document similarity—often performed as well as or better than manually curated semantic taxonomies. Replacing the native hierarchy of an HTC model with a synthetic XML-style one occasionally improved performance, suggesting that data-driven clustering captures relationships that manual trees might miss.
Results
XML models proved to be highly versatile, effectively treating HTC tasks as simplified versions of their native problems.
As shown in the results, CascadeXML emerged as a powerhouse, achieving the best or second-best results across almost all HTC datasets. For instance, on the NY-Times dataset, CascadeXML achieved an R-Precision of 84.61, surpassing the specialized HTC model HGCLR (84.22). Conversely, while HTC models performed well on their home turf, they were unable to complete training on the largest XML datasets due to resource constraints.
Conclusion
This research reveals that XML models are "secretly" excellent hierarchical classifiers. Their ability to infer robust relational information from raw data allows them to outperform models that rely on hand-crafted taxonomies. For the HTC community, these findings highlight a desperate need for more efficient label-handling techniques to bridge the gap toward modern, large-scale datasets. Future work should explore hybrid approaches that combine the strengths of semantic knowledge with the scalability of data-driven clustering.
Summary updated