Abstract: Multi-label classification (MC) is a standard machine learning problem in which a data point can be associated with a set of classes. A more challenging scenario is given by hierarchical multi-label classification (HMC) problems, in which every prediction must satisfy a given set of hard constraints expressing subclass relationships between classes. In this paper, we propose C-HMCNN(h), a novel approach for solving HMC problems, which, given a network h for the underlying MC problem, exploits the hierarchy information in order to produce predictions coherent with the constraints and to improve performance. Furthermore, we extend the logic used to express HMC constraints in order to be able to specify more complex relations among the classes and propose a new model CCN(h), which extends C-HMCNN(h) and is again able to satisfy and exploit the constraints to improve performance. We conduct an extensive experimental analysis showing the superior performance of both C-HMCNN(h) and CCN(h) when compared to state-of-the-art models in both the HMC and the general MC setting with hard logical constraints.
Introduction
In many real-world multi-label classification tasks, labels are not independent but governed by fixed logical relationships. The most prevalent structure is a hierarchy: for instance, in medical diagnostics, a specific condition like "stomach ulcer" implies the broader category of "gastrointestinal disease." Standard neural networks typically treat these labels as independent outputs, which frequently results in logically inconsistent predictions—such as assigning a high probability to a specific subclass while assigning a low probability to its mandatory superclass.
While traditional methods rely on post-processing to rectify these inconsistencies, such "band-aid" solutions prevent the model from learning the underlying data structure during training. This paper introduces C-HMCNN(h) and CCN(h), architectures designed to embed hard logical constraints directly into the neural network, ensuring consistency by construction and improving overall predictive performance.
Methodology
The proposed approach integrates logical constraints through two primary mechanisms: a specialized architectural layer and a tailored loss function.
The Constraint Layer (CM)
Rather than using a standard output layer, the authors introduce a constraint layer that enforces logical rules using max and min operations. In a hierarchical setting, the probability of a parent class is calculated as the maximum of its own predicted probability and the probabilities of all its descendants. This architectural choice ensures that the output signal inherently respects the hierarchy: a parent class score will always be at least as high as any of its children's scores.
Max Constraint Loss (CLoss)
Standard binary cross-entropy loss can provide misleading gradients when constraints are violated. To address this, the authors developed the Max Constraint Loss (CLoss). This loss function is mathematically formulated to ensure that gradients always move model weights toward a state that satisfies the logical constraints. It effectively teaches the model to "delegate" a prediction to a more specific subclass if the evidence is strong, rewarding the synergy between related labels.
Generalizing to Complex Logic: CCN(h)
To handle more complex logical relationships beyond simple hierarchies (e.g., "if A and B, then C"), the authors propose the Constrained Classification Network (CCN). This model utilizes concepts from logic programming, specifically stratified negation, to handle complex rules iteratively and avoid circular logic.
Experimental Results
The models were evaluated across a diverse set of benchmarks, including 20 HMC datasets (genomics, medical imaging, text categorization) and 16 LCMC datasets featuring complex logical rules extracted from standard multi-label data.
Key findings include:
- Superior Performance: C-HMCNN and CCN consistently outperformed state-of-the-art models like HMCN-F and Clus-Ens, achieving the best average ranking (1.25) across hierarchical datasets.
- Perfect Consistency: The proposed models achieved a 0% logical violation rate across all test cases without requiring inference-time adjustments or post-processing.
- Efficiency: The GPU-based implementation allows for parallelized logical checks, maintaining low inference times (1.6ms to 3.9ms per batch) even with massive taxonomies.
- Scalability: The number of parameters in C-HMCNN is independent of hierarchy depth, making it significantly more efficient than previous hierarchical models.
Discussion and Conclusion
By shifting from post-processing to architectural enforcement, C-HMCNN(h) and CCN(h) provide a robust framework for multi-label classification under hard constraints. The models not only guarantee logical consistency but also leverage the structural information to improve accuracy.
However, some limitations remain. The current CCN model requires rules to be "stratified" to avoid circular definitions involving negations. Additionally, the current work focuses on hard constraints on output labels; future research could explore soft constraints or rules linking input features to outputs. Nevertheless, the results demonstrate that integrating logic into the heart of neural networks is a powerful strategy for structured classification problems.
Summary updated