Abstract
Hierarchical classification tasks often encounter the "small training set" problem, where nodes deep within a taxonomy lack sufficient data for robust model training. This paper introduces a framework based on Label Distribution Learning (LDL) to address this challenge. Instead of traditional hard labels, the proposed method utilizes the hierarchical structure to generate a "soft" distribution of related sibling labels, providing additional supervision and preventing overfitting. Experimental results across five diverse datasets show that this approach significantly outperforms existing baselines, especially in scenarios with limited training data.
Introduction
In hierarchical classification, labels are structured in a tree or Directed Acyclic Graph (e.g., Electronics > Audio > Headphones). Most existing systems employ local classifiers at each branch. However, as the hierarchy deepens, the number of training examples per label typically decreases. For instance, in the LSHTC-small dataset, the average number of samples per label is fewer than six. In such sparse data environments, local classifiers often overfit, treating closely related sibling classes as unrelated negative examples. This paper argues that the structural relationships between labels contain valuable information that can be used to regularize training.
Methods
The core of the proposed framework is the shift from predicting a single "correct" label to predicting a Label Distribution.
Label-to-Distribution Transformation
Since ground-truth data typically provides only a single label, the authors transform these into distributions based on the Number of Common Nodes with the True Path (NCNTP). If an instance belongs to a specific node, its siblings sharing more ancestors with that node are assigned higher probability weights (description degrees) in the training distribution.
Virtual Nodes and Optimization
To handle instances that do not belong to any specific local sub-branch, "virtual nodes" are introduced. This allows local classifiers to distinguish between relevant sub-classes and other content within the hierarchy. The model employs a maximum entropy approach, minimizing the Jeffrey’s divergence between the predicted and structural ground-truth distributions using the L-BFGS algorithm.
Path-Score Prediction
During inference, the model calculates a Path Score that normalizes probabilities by path length. This prevents the system from unfairly penalizing deeper, more specific nodes, which naturally accumulate lower raw probabilities in a top-down traversal.
Key Results
The LDL-based method was evaluated against state-of-the-art approaches including Flat-SVM, Top-Down (TD), and HierCost on datasets such as CLEF, IPC, LSHTC-small, and DMOZ.
Performance and Data Sparsity Resilience
The proposed method consistently achieved superior results. On the LSHTC-small dataset, it reached a Micro-F1 of 50.51 and a Tree-induced Error (TE) of 3.07, outperforming HierCost (49.46 F1, 3.45 TE).
The most significant advantage was observed when training data was artificially reduced. As shown in the figure below, while standard models' performance degraded rapidly as data became scarcer (down to 1/40th of the original set), the LDL-based model maintained high accuracy and low error rates.
On the large-scale DMOZ-2012 dataset, the method achieved a Micro-F1 of 55.30, further validating its effectiveness on massive taxonomies.
Conclusion
By treating the hierarchy's structure as a prior distribution, this paper effectively addresses the data sparsity problem in hierarchical classification. The use of Label Distribution Learning allows models to learn from the "neighborhood" of a class, providing a form of structural regularization. Future work includes extending this framework to non-tree hierarchies and optimizing computational complexity for real-time applications with millions of labels.
Summary updated