Emergent symbolic language based deep medical image classification

Aritra Chowdhury, Alberto Santamaría-Pang, James R. Kubricht, Peter H. Tu

Introduction

The "black box" nature of deep learning remains a significant barrier to the clinical adoption of AI. While convolutional neural networks (CNNs) excel at identifying patterns in medical imagery, their decision-making processes rely on high-dimensional latent representations that are unintelligible to humans. This paper proposes a novel framework that replaces these continuous representations with an emergent symbolic language, bridging the gap between the performance of connectionism and the interpretability of symbolic logic.

Methodology: The Sender-Receiver Architecture

The authors frame medical image classification as a communication game between two neural agents. The architecture consists of three core components:

  1. Sender Network: A deep neural network (DNN) that extracts features from the input image.
  2. Symbol Generator: A bottleneck layer that converts continuous features into discrete symbols using a Gumbel-Softmax estimator. This allows the network to remain end-to-end differentiable while forcing it to communicate via a discrete "alphabet."
  3. Receiver Network: A network that processes these symbols to generate the final diagnostic classification.
The emergent symbolic framework (bottom) introduces a discrete symbol generator between sender and receiver networks, unlike traditional continuous DNNs (top).

To ensure the symbols are clinically meaningful, the researchers use Neuron Conductance to attribute symbol generation to specific physical features in the input images. This effectively allows clinicians to "trace back" a model's decision to observable biological markers.

Experimental Evaluation

The framework was validated across two diverse medical imaging domains:

  • Immunofluorescence Cell Markers: Classifying colon tissue samples into protein marker categories (CD3, CD20, CD68, and Claudin1) based on 28 statistical and shape-based features.
  • CheXpert (Pleural Effusion): Identifying the presence of fluid in the lungs using chest X-rays.
Representative samples from the immunofluorescence cell marker and CheXpert chest X-ray datasets.

Results and Discussion

The symbolic models achieved performance comparable to or better than standard baselines. In the CheXpert task, the symbolic model reached 77.35% accuracy, outperforming the 73.07% baseline.

Task Baseline Accuracy Symbolic Accuracy
CheXpert (Pleural Effusion) 73.07% 77.35%
Immune Cell Markers 98.20% 97.07%

Key Findings

  • Regularization through Discretization: The performance boost in the CheXpert task suggests that the symbolic bottleneck acts as a regularizer, forcing the model to discard noise and focus on robust, generalized features.
  • Vocabulary Efficiency: Despite being provided with a vocabulary of 100 symbols, the models naturally converged on a small subset. This suggests that complex medical phenotypes can be represented by a very compact symbolic alphabet.
  • Phenotypical Grounding: Symbols were found to correlate with specific disease presentations, such as the severity of lung effusion. Specific symbols even emerged to flag cases of high uncertainty or potential annotation errors, which could be used to trigger human expert review.

Conclusion

This work demonstrates that forcing neural networks to communicate via discrete symbols can enhance interpretability without compromising diagnostic accuracy. By making the latent space human-readable, the framework provides a pathway toward more transparent medical AI. Future research will focus on generating multi-symbol "sentences" and grounding these symbols in natural language to produce automated, human-readable radiology reports.

Summary updated

BibTeX
@inproceedings{chowdhury2021emergent,
  author = {Aritra Chowdhury and Alberto Santamaría-Pang and James R. Kubricht and Peter H. Tu},
  title = {{Emergent symbolic language based deep medical image classification}},
  booktitle = {ISBI},
  year = {2021},
  doi = {10.1109/isbi48211.2021.9434073},
}
Figure