Introduction
Hierarchical Text Classification (HTC) is the task of assigning documents to labels organized within a structured taxonomy, such as news categories or legal codes. Traditionally, deep learning approaches for HTC have been categorized into local and global methods. Local approaches train separate classifiers for different hierarchy levels, which often leads to error propagation. Global approaches attempt to model the entire hierarchy simultaneously, typically using Graph Convolutional Networks (GCNs) to encode label relationships.
However, global models face a significant scalability bottleneck. As taxonomies grow to include thousands of labels, the parameter count of graph-based encoders increases dramatically. This inefficiency stems from processing the entire hierarchy for every document, regardless of the relevance of most labels to the specific text.
Methodology: The HiDEC Framework
To address these challenges, the researchers introduced HiDEC (Hierarchy DECoder), a framework that reformulates HTC as a sub-hierarchy sequence generation task. Instead of treating the problem as a standard multi-label classification against a static tree, HiDEC generates a sequence representing only the relevant paths from the root to the target labels.
Recursive Hierarchy Decoding
The core of HiDEC is a recursive decoding process. Starting from the root node, the model predicts which child labels are relevant for each active parent label. This process continues level-by-level until leaf nodes are reached or an [END] token is generated. This approach ensures that the model only focuses on the sparse subset of labels pertinent to the input text.
Technical Innovations
- Hierarchy-Aware Masking: Rather than using a heavy graph encoder, HiDEC incorporates structural constraints directly into the Transformer's self-attention mechanism. A custom mask ensures that labels only attend to their ancestors, effectively encoding the hierarchy within the attention scores.
- Text-Hierarchy Attention: This component dynamically aligns document features from a text encoder (e.g., BERT) with the generated label sequence, allowing the model to attend to specific text segments relevant to different hierarchy levels.
- Parse Tree Notation: Target labels are represented as sequences using nested parentheses (e.g.,
(Root (Parent (Child)))), enabling the decoder to learn structural dependencies naturally.
Experimental Evaluation
HiDEC was evaluated on three benchmark datasets:
- RCV1-v2: News dataset (103 labels, depth 4).
- NYT: New York Times dataset (166 labels, depth 8).
- EURLEX57K: Large-scale legal dataset (4,271 labels, depth 5), used to test scalability.
Results and Discussion
HiDEC achieved state-of-the-art (SOTA) performance across all benchmarks, outperforming models like HGCLR and HiMatch in both Micro-F1 and Macro-F1 scores.
Efficiency and Scalability
The most significant advantage of HiDEC is its parameter efficiency. On the EURLEX57K dataset, HiDEC achieved superior accuracy while utilizing only 21 million parameters, compared to the 6,211 million parameters required by the previous SOTA (HiMatch). This represents a 295x reduction in model size. Furthermore, HiDEC's parameters scale linearly with the number of labels, making it highly suitable for extreme-scale taxonomies.
Structural Insights
Analysis indicates that HiDEC's performance gains are most pronounced at deeper levels of the hierarchy. Interestingly, the researchers found that removing standard residual connections from the Transformer decoder improved performance, suggesting that residual paths may dilute essential hierarchical information in this specific context.
Conclusion
HiDEC demonstrates that treating hierarchical text classification as a sequence generation task significantly improves both performance and efficiency. By focusing on relevant sub-hierarchies and utilizing hierarchy-aware attention, HiDEC overcomes the scalability limitations of traditional global models. Future work will explore the application of HiDEC to extremely large-scale taxonomies with hundreds of thousands of labels.
Summary updated