Hierarchical Text Classification (HTC) vs. eXtreme Multilabel Classification (XML): Two Sides of the Same Medal

Nerijus Bertalis, Paul Granse, Ferhat Gül, Florian Hauss, Leon Menkel, David Schüler, Tom Speier, Lukas Galke, Ansgar ScherparXiv 2024 · arXiv · OpenReview

Abstract

Assigning a subset of labels from a fixed pool to a given input text is a fundamental classification problem with diverse applications, including recommender systems. Currently, two separate research streams address this: Hierarchical Text Classification (HTC), which focuses on smaller label pools (hundreds) with semantic hierarchies, and eXtreme Multi-Label Text Classification (XML), which handles massive label pools (up to millions) without explicit hierarchies. This paper investigates the bidirectional transferability of state-of-the-art models between these domains. We find that while XML models like CascadeXML and XR-Transformer excel on HTC tasks—often outperforming specialized HTC models—HTC models like HBGL and HGLCR struggle to scale to XML datasets. Our results suggest that the artificial hierarchies constructed by XML models are highly effective even when semantic taxonomies are available.

Introduction

In Natural Language Processing (NLP), multi-label classification is typically divided into two silos. HTC datasets feature human-curated taxonomies (e.g., library classifications), while XML datasets deal with such extreme scale that models must build their own internal "artificial" trees for computational efficiency. Because these fields use different evaluation metrics (F1-scores for HTC vs. Precision@k for XML), they rarely interact. This study breaks down these barriers to ask whether XML models can solve HTC problems and if HTC models can scale to the "extreme" level.

Methodology

To facilitate a fair comparison, the researchers established a bidirectional transfer framework:

  1. Metric Harmonization: The authors introduced R-Precision as a primary metric. Unlike Precision@k or F1, R-Precision evaluates the top $r$ predictions, where $r$ is the actual number of ground-truth labels for a document, providing a threshold-independent view of ranking quality.
  2. Dataset Conversion:
    • XML on HTC: The semantic hierarchy was "collapsed," treating labels as a flat set and allowing XML models to build internal clusters.
  3. HTC on XML: Synthetic hierarchies were generated for XML datasets using Positive Instance Feature Aggregation (PIFA) combined with balanced k-means.
  4. Model Selection: The study compared transformer-based models: HGCLR and HBGL (HTC domain) against XR-Transformer and CascadeXML (XML domain).

Datasets

The study utilized six benchmark datasets spanning both domains. The primary distinction lies in the label count and the presence of a predefined hierarchy.

Table 1: Statistics for HTC and XML datasets, highlighting the massive scale difference in label counts.

Key Insights

The Scalability Wall

A significant finding was the inability of HTC models to handle extreme scales. Architectures like HGCLR and HBGL often rely on $O(n^2)$ components, such as dense attention or complex graph encoders. These models frequently crashed or exceeded memory limits when faced with the 670,000 labels of the Amazon-670K dataset.

Data-Driven vs. Semantic Hierarchies

Surprisingly, XML-style artificial hierarchies—built on label frequency and document similarity—often performed as well as or better than manually curated semantic taxonomies. Replacing the native hierarchy of an HTC model with a synthetic XML-style one occasionally improved performance, suggesting that data-driven clustering captures relationships that manual trees might miss.

Results

XML models proved to be highly versatile, effectively treating HTC tasks as simplified versions of their native problems.

Table 2: Performance comparison across three HTC datasets. XML models (bottom section) frequently match or exceed specialized HTC models (top section).

As shown in the results, CascadeXML emerged as a powerhouse, achieving the best or second-best results across almost all HTC datasets. For instance, on the NY-Times dataset, CascadeXML achieved an R-Precision of 84.61, surpassing the specialized HTC model HGCLR (84.22). Conversely, while HTC models performed well on their home turf, they were unable to complete training on the largest XML datasets due to resource constraints.

Conclusion

This research reveals that XML models are "secretly" excellent hierarchical classifiers. Their ability to infer robust relational information from raw data allows them to outperform models that rely on hand-crafted taxonomies. For the HTC community, these findings highlight a desperate need for more efficient label-handling techniques to bridge the gap toward modern, large-scale datasets. Future work should explore hybrid approaches that combine the strengths of semantic knowledge with the scalability of data-driven clustering.

Summary updated

Abstract

Assigning a subset of labels from a fixed pool of labels to a given input text is a text classification problem with many real-world applications, such as in recommender systems. Two separate research streams address this issue. Hierarchical Text Classification (HTC) focuses on datasets with smaller label pools of hundreds of entries, accompanied by a semantic label hierarchy. In contrast, eXtreme Multi-Label Text Classification (XML) considers very large label pools with up to millions of entries, in which the labels are not arranged in any particular manner. However, in XML, a common approach is to construct an artificial hierarchy without any semantic information before or during the training process. Here, we investigate how state-of-the-art models from one domain perform when trained and tested on datasets from the other domain. The HBGL and HGLCR models from the HTC domain are trained and tested on the datasets Wiki10-31K, AmazonCat-13K, and Amazon-670K from the XML domain. On the other side, the XML models CascadeXML and XR-Transformer are trained and tested on the datasets Web of Science, The New York Times Annotated Corpus, and RCV1-V2 from the HTC domain. HTC models, on the other hand, are not equipped to handle the size of XML datasets and achieve poor transfer results. The code and numerous files that are needed to reproduce our results can be obtained from https://github.com/FloHauss/XMC_HTC
BibTeX
@misc{bertalis2024hierarchical,
  author = {Nerijus Bertalis and Paul Granse and Ferhat Gül and Florian Hauss and Leon Menkel and David Schüler and Tom Speier and Lukas Galke and Ansgar Scherp},
  title = {{Hierarchical Text Classification (HTC) vs. eXtreme Multilabel Classification (XML): Two Sides of the Same Medal}},
  year = {2024},
  eprint = {2411.13687},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.13687},
}
Figure