Closing the Neural-Symbolic Cycle: Knowledge Extraction, User Intervention and Distillation from Convolutional Neural Networks

Kwun Ho Ngan, James Phelan, Esma Mansouri-Benssassi, Joe Townsend, Artur S. d'Avila GarcezNesy 2023

Abstract

Deep learning models like Convolutional Neural Networks (CNNs) excel at medical image analysis but remain "black boxes," lacking transparency for clinical use. This paper introduces a "neural-symbolic cycle" that extracts symbolic logic from a CNN, allows medical experts to refine these rules, and distills the validated knowledge into a compact student network. Results show that expert intervention ensures clinical relevance and enables extreme model compression while maintaining high diagnostic accuracy.

Introduction

In high-stakes medical fields like radiology, AI models must be interpretable. Existing Explainable AI (XAI) methods often rely on heatmaps that show which pixels a model prioritized, but they fail to explain the underlying logic. Furthermore, models can "cheat" by picking up on irrelevant features, such as a patient's collarbone, to predict lung conditions. This research addresses the gap between high performance and clinical transparency by creating a feedback loop between neural networks and human experts.

The Neural-Symbolic Cycle

The researchers developed a four-stage cycle to bridge the gap between neural performance and symbolic interpretability:

  1. Knowledge Extraction: Symbolic rules, in the form of a decision tree, are extracted from the final convolutional layer of a trained VGG-16 "teacher" network using the ERIC framework.
  2. Concept Association: Internal filters (kernels) are mapped to anatomical regions (e.g., heart silhouette) using a YOLOv5 segmentation model and correlated with radiomics features like texture and entropy.
  3. User Intervention: A medical professional reviews the rules, discarding irrelevant ones and selecting kernels clinically relevant to diagnosing pleural effusion (fluid in the lungs).
  4. Knowledge Distillation: A "student" CNN is trained to mimic the logic of the refined rules rather than just raw data labels.
Overview of the Neural-Symbolic Cycle, showing the flow from model training to knowledge extraction, expert intervention, and final model re-training via a teacher-student network.

Key Insights and Results

The study utilized the CheXpert dataset for pleural effusion detection and the NIH dataset for anatomical segmentation. Key findings include:

  • Semantic Kernels: Specific CNN filters naturally gravitate toward anatomical landmarks. For example, kernel "QN" was found to be highly correlated with the cardiac silhouette.
  • Clinical Alignment: The model's use of entropy in the heart region to detect fluid mimics the "silhouette sign," a classic radiological indicator where fluid obscures the heart's border.
  • Performance Retention: Expert refinement (removing "noisy" rules) maintained high accuracy. The initial rules had 94.5% accuracy, while the expert-refined set reached 94.8%.
  • Extreme Compression: The student model, stripped down to just four key kernels (a 90% reduction in the bottleneck layer), maintained a diagnostic accuracy of 93.8%.
Comparison of the (a) Initial Rule Set and (b) Clinically Relevant Rule Set after expert intervention, highlighting the replacement of irrelevant features and maintained performance metrics.

Conclusion and Future Work

The neural-symbolic cycle demonstrates that clinical pruning of AI logic does not necessarily degrade performance and can lead to highly efficient, auditable models. While currently tested on a small subset of data, future work aims to scale this approach to larger datasets and automate concept association to provide a fully transparent knowledge transfer trail for AI in healthcare.

Summary updated

BibTeX
@inproceedings{ngan2023closing,
  author = {Kwun Ho Ngan and James Phelan and Esma Mansouri-Benssassi and Joe Townsend and Artur S. d'Avila Garcez},
  title = {{Closing the Neural-Symbolic Cycle: Knowledge Extraction, User Intervention and Distillation from Convolutional Neural Networks}},
  booktitle = {Nesy},
  year = {2023},
}
Figure