Emergent Symbolic Language Based Deep Medical Image Classification

Aritra Chowdhury, Alberto Santamaría-Pang, James R. Kubricht, Peter H. Tu

TL;DR

This paper introduces a novel framework for medical image classification that replaces the "black box" latent representations of deep learning with an emergent symbolic language. By forcing neural networks to communicate via discrete symbols, the authors achieve classification performance that matches or exceeds traditional models while providing a clear, interpretable pathway to understand the model's decision-making process.

What Problem Does This Paper Address?

While deep learning has revolutionized medical image analysis, its adoption in clinical settings is hindered by a lack of interpretability. Modern convolutional neural networks (CNNs) rely on continuous latent representations—essentially high-dimensional "word salad" that humans cannot parse. This makes it nearly impossible for a clinician to understand why a model classified a scan as malignant versus benign.

Historically, "Good Old Fashioned AI" (GOFAI) used symbolic logic which was highly interpretable but too inflexible to handle complex raw data like images. This paper seeks to bridge that gap by allowing a symbolic language to "emerge" naturally within a deep learning architecture, combining the representational power of connectionism with the explainability of symbols.

Methods

The authors propose an Emergent Symbolic Classification framework based on multi-agent communication games. The architecture consists of three main components:

  1. Sender Network: A DNN (or CNN) that extracts features from the input image.
  2. Symbol Generator: A bottleneck layer that converts continuous features into discrete symbols. To keep the network end-to-end differentiable, they use a Gumbel-Softmax estimator, which allows for categorical sampling during the forward pass while maintaining gradient flow for backpropagation.
  3. Receiver Network: A DNN that takes these discrete symbols as input and produces the final classification.
The emergent symbolic framework (bottom) introduces a symbol generator between the sender and receiver networks, unlike traditional "black box" DNNs (top).

To explain what these symbols actually mean, the authors use Neuron Conductance. This technique attributes the importance of specific input features to the generation of a particular symbol, effectively "tracing back" the symbol to physical characteristics in the image.

Datasets

The framework was evaluated on two distinct medical tasks:

  • Immunofluorescence Cell Markers: A dataset of colon tissue samples. The task involves classifying 28 statistical and shape-based features into four protein marker categories (CD3, CD20, CD68, and Claudin1).
  • CheXpert (Pleural Effusion): A subset of the large-scale CheXpert dataset containing chest X-rays. The model was tasked with identifying the presence or absence of pleural effusion (fluid in the lungs).
Examples of the datasets: (Left) Multi-channel immune cell markers; (Right) Frontal and lateral chest X-rays.

Key Insights and Ideas

  • Symbols as Regularizers: Surprisingly, the symbolic model outperformed the baseline CNN on the CheXpert dataset. The authors suggest that the discrete symbolic bottleneck acts as a form of regularization, preventing the model from over-fitting on noise and forcing it to learn more generalized, robust features.
  • Vocabulary Efficiency: Even though the models were given a vocabulary size of 100, they naturally converged on using only a small handful of unique symbols to solve the tasks. This suggests that medical categories can be represented by a very compact "alphabet."
  • Phenotypical Mapping: The symbols weren't just random labels; they mapped to specific physical presentations. For example, different symbols emerged for different "extents" of lung effusion, and one specific symbol (Symbol 88) appeared to capture cases where the model was uncertain or where human annotations might be incorrect.

Key Results

The emergent symbolic approach proved to be highly competitive with standard "black box" baselines:

Experiment Accuracy (%) F1-score
CheXpert Baseline 73.07 0.677
CheXpert Symbolic 77.35 0.697
Immune Cell Baseline 98.20 0.982
Immune Cell Symbolic 97.07 0.971

In the CheXpert task, the symbolic model actually provided a ~4% accuracy boost. Qualitatively, the researchers used neuron conductance to show that the symbols were picking up relevant biological information (e.g., specific protein marker intensities) rather than arbitrary patterns.

Neuron conductance values show how specific input features contribute to the generation of unique symbols.

Limitations and Future Work

The current study focuses on a "single symbol" framework, where one symbol represents the entire image. The authors acknowledge that this is just the beginning. Future work aims to:

  • Sentence Generation: Use LSTMs or Transformers to allow the agents to generate sequences of symbols (sentences).
  • Natural Language Grounding: Interface these emergent symbols with actual human language, such as radiology reports, to create AI that can "write" its own justifications in plain English.
  • Active Learning: Use "uncertainty symbols" (like Symbol 88 in the CheXpert test) to flag difficult cases for human expert review, creating a more collaborative diagnostic loop.

Summary updated

BibTeX
@inproceedings{chowdhury2021emergent,
  author = {Aritra Chowdhury and Alberto Santamaría-Pang and James R. Kubricht and Peter H. Tu},
  title = {{Emergent Symbolic Language Based Deep Medical Image Classification}},
  booktitle = {ISBI},
  year = {2021},
  doi = {10.1109/isbi48211.2021.9434073},
}
Figure