Abstract
Neural networks are powerful predictors but lack transparency, which is critical in fields like medicine and law. This paper proposes an entropy-based neural network layer that bridges the gap between high-performance deep learning and human-interpretable logic. By regularizing the entropy of concept relevance, the model is forced to rely on a sparse set of human-understandable concepts. This architecture allows for the distillation of decision-making into concise First-Order Logic (FOL) formulas without compromising accuracy. Evaluated across clinical, political, and image datasets, the approach demonstrates superior performance and fidelity compared to traditional white-box models.
Introduction: The Interpretability Challenge
While neural networks excel at predictive tasks, their "black-box" nature remains a significant barrier in safety-critical applications. Existing Explainable AI (XAI) methods often fall into two categories: post-hoc explanations, which identify important features but lack logical structure, and white-box models, which are transparent but struggle with high-dimensional data. This research addresses the need for models that are explainable-by-design. By leveraging concept-based learning, the model identifies high-level, human-understandable attributes (e.g., "has wings") and composes them into formal logic to reach a conclusion.
Methodology: Entropy-based Logic Distillation
The core innovation is the Entropy-based Linear Layer, which structures the network to process data through a "concept space."
The process involves three key stages:
- Concept Bottleneck: A standard backbone (e.g., ResNet) maps raw input pixels to human-understandable attributes.
- Entropy Regularization: The specialized layer assigns relevance scores ($\alpha$) to concepts. An entropy-based loss term penalizes high-entropy distributions, forcing the model to rely on a sparse, essential subset of concepts. This is more effective than standard L1 regularization at shrinking irrelevant concept scores to zero.
- Logic Extraction: The model’s behavior is mapped onto a truth table, from which First-Order Logic (FOL) formulas in Disjunctive Normal Form (DNF) are extracted. This provides both local (example-level) and global (class-level) explanations.
Experimental Evaluation
The approach was tested across four diverse domains to demonstrate its versatility:
- MIMIC-II: Clinical data used to predict patient recovery post-ICU admission.
- V-Dem: A political science dataset used to identify types of democracies based on latent regime characteristics.
- MNIST (Even/Odd): Handwritten digits repurposed to explain parity (even vs. odd) based on digit labels.
- CUB (Caltech-UCSD Birds-200-2011): Fine-grained bird classification using 312 binary concept attributes.
Key Results
- Performance: The Entropy-based network consistently matched or exceeded the accuracy of black-box models. For instance, on the V-Dem dataset, it achieved 94.51% accuracy, outperforming Decision Trees (85.61%) and Bayesian Rule Lists (91.23%).
- Fidelity and Conciseness: The extracted logic formulas showed remarkably high fidelity—often exceeding 99%—meaning they accurately represent the underlying neural network's logic. Furthermore, the rules remained concise, typically involving fewer than seven concepts, aligning with human cognitive limits.
Conclusion and Future Directions
The proposed entropy-based framework demonstrates that explainability and high performance are not mutually exclusive. By building interpretability into the architecture, the model provides clear, logical justifications for its decisions. While currently dependent on concept annotations, future work may integrate unsupervised concept extraction. Expanding this approach to Transformers or Graph Neural Networks could bring similar transparency to fields like Natural Language Processing and drug discovery.
The implementation is available via the "entropy-lens" library on GitHub.
Summary updated