Hierarchical Classification Based on Label Distribution Learning

Changdong Xu, Xin Geng

Abstract

Hierarchical classification tasks often encounter the "small training set" problem, where nodes deep within a taxonomy lack sufficient data for robust model training. This paper introduces a framework based on Label Distribution Learning (LDL) to address this challenge. Instead of traditional hard labels, the proposed method utilizes the hierarchical structure to generate a "soft" distribution of related sibling labels, providing additional supervision and preventing overfitting. Experimental results across five diverse datasets show that this approach significantly outperforms existing baselines, especially in scenarios with limited training data.

Introduction

In hierarchical classification, labels are structured in a tree or Directed Acyclic Graph (e.g., Electronics > Audio > Headphones). Most existing systems employ local classifiers at each branch. However, as the hierarchy deepens, the number of training examples per label typically decreases. For instance, in the LSHTC-small dataset, the average number of samples per label is fewer than six. In such sparse data environments, local classifiers often overfit, treating closely related sibling classes as unrelated negative examples. This paper argues that the structural relationships between labels contain valuable information that can be used to regularize training.

Methods

The core of the proposed framework is the shift from predicting a single "correct" label to predicting a Label Distribution.

Label-to-Distribution Transformation

Since ground-truth data typically provides only a single label, the authors transform these into distributions based on the Number of Common Nodes with the True Path (NCNTP). If an instance belongs to a specific node, its siblings sharing more ancestors with that node are assigned higher probability weights (description degrees) in the training distribution.

Comparison of traditional single-label assignment versus the proposed label distribution learning approach.

Virtual Nodes and Optimization

To handle instances that do not belong to any specific local sub-branch, "virtual nodes" are introduced. This allows local classifiers to distinguish between relevant sub-classes and other content within the hierarchy. The model employs a maximum entropy approach, minimizing the Jeffrey’s divergence between the predicted and structural ground-truth distributions using the L-BFGS algorithm.

Path-Score Prediction

During inference, the model calculates a Path Score that normalizes probabilities by path length. This prevents the system from unfairly penalizing deeper, more specific nodes, which naturally accumulate lower raw probabilities in a top-down traversal.

Key Results

The LDL-based method was evaluated against state-of-the-art approaches including Flat-SVM, Top-Down (TD), and HierCost on datasets such as CLEF, IPC, LSHTC-small, and DMOZ.

Performance and Data Sparsity Resilience

The proposed method consistently achieved superior results. On the LSHTC-small dataset, it reached a Micro-F1 of 50.51 and a Tree-induced Error (TE) of 3.07, outperforming HierCost (49.46 F1, 3.45 TE).

The most significant advantage was observed when training data was artificially reduced. As shown in the figure below, while standard models' performance degraded rapidly as data became scarcer (down to 1/40th of the original set), the LDL-based model maintained high accuracy and low error rates.

Performance comparison across varying training set sizes, highlighting the model's resilience to data sparsity.

On the large-scale DMOZ-2012 dataset, the method achieved a Micro-F1 of 55.30, further validating its effectiveness on massive taxonomies.

Conclusion

By treating the hierarchy's structure as a prior distribution, this paper effectively addresses the data sparsity problem in hierarchical classification. The use of Label Distribution Learning allows models to learn from the "neighborhood" of a class, providing a form of structural regularization. Future work includes extending this framework to non-tree hierarchies and optimizing computational complexity for real-time applications with millions of labels.

Summary updated

Abstract

Hierarchical classification is a challenging problem where the class labels are organized in a predefined hierarchy. One primary challenge in hierarchical classification is the small training set issue of the local module. The local classifiers in the previous hierarchical classification approaches are prone to over-fitting, which becomes a major bottleneck of hierarchical classification. Fortunately, the labels in the local module are correlated, and the siblings of the true label can provide additional supervision information for the instance. This paper proposes a novel method to deal with the small training set issue. The key idea of the method is to represent the correlation among the labels by the label distribution. It generates a label distribution that contains the supervision information of each label for the given instance, and then learns a mapping from the instance to the label distribution. Experimental results on several hierarchical classification datasets show that our method significantly outperforms other state-of-theart hierarchical classification approaches.
BibTeX
@inproceedings{xu2019hierarchical,
  author = {Changdong Xu and Xin Geng},
  title = {{Hierarchical Classification Based on Label Distribution Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.1609/aaai.v33i01.33015533},
}
Figure