Introduction
Hierarchical multi-label text classification (HMLTC) is a critical task in domains like biomedicine, where documents are tagged with labels from complex taxonomies. For instance, a document labeled "Lung Adenocarcinoma" inherently implies labels like "Carcinoma" and "Cancer." The standard One-vs-Rest (OVR) approach, which trains independent binary classifiers for each label, fails to capture these dependencies and becomes computationally prohibitive as the number of labels grows. This paper proposes a novel method to incorporate hierarchical structural information directly into the neural network's initial state by pre-configuring the weights of the final hidden layer to reflect known label co-occurrences.
Methodology
The proposed approach utilizes a Convolutional Neural Network (CNN) architecture optimized for text classification. The core innovation is the "Smart Initialization" of the final dense layer's weights.
Smart Initialization (INIT-A & INIT-B)
Rather than using random initialization, the authors initialize specific neurons in the final hidden layer to represent label co-occurrence patterns observed in the training data.
- INIT-A: Rows in the weight matrix corresponding to frequent label sets (e.g., {$y_1, y_3$}) are initialized with a positive value $\omega$ for those labels and zero for others.
- INIT-B: Similar to INIT-A, but instead of zeroing out non-co-occurring labels, it uses a different baseline (indicated by # in the diagrams).
The value $\omega$ is determined using the Glorot and Bengio (2010) normalized initialization strategy to maintain stable signal variance during the early stages of training.
Hierarchical Logic and Post-processing
The initialization naturally encodes hypernymy; if a subclass label is present, the superclass label is also activated. To further refine predictions, the authors evaluated two logic-based post-processing steps: transitive correction (forcing superclasses to be positive if a subclass is predicted) and retractive correction (forcing subclasses to be negative if a superclass is predicted negative).
Experimental Setup
The method was evaluated on two biomedical datasets:
- Hallmarks of Cancer: 37 classes representing biological properties of cancer.
- Chemical Exposure Taxonomy: 32 classes focused on chemical risk assessment.
Evaluations were conducted at both the document level (high label density) and sentence level (low label density).
Results and Discussion
The INIT-A strategy emerged as the superior performer, particularly in document-level classification. In the Hallmarks of Cancer task, INIT-A achieved an $F_1$-score of 75.1%, significantly outperforming the standard multi-label baseline (71.3%) and the OVR baseline (62.1%).
The performance boost was primarily driven by an increase in recall. By "priming" the network to recognize related labels together, the model became less conservative in assigning hierarchical labels. The method proved more effective at the document level, where label overlap is more frequent, providing a stronger co-occurrence signal for the initialization to exploit.
Conclusion
By "baking" structural priors into the initial weights of a neural network, this research demonstrates a simple yet effective way to improve HMLTC without increasing model complexity or training time. While the method's efficacy depends on data density and label co-occurrence frequency, it offers a promising alternative to more complex recursive or loss-constrained hierarchical models.
Summary updated