Towards Emergent Language Symbolic Semantic Segmentation and Model Interpretability

Alberto Santamaría-Pang, James Kubricht, Aritra Chowdhury, Chitresh Bhushan, Peter H. Tu

Introduction

Deep learning models, particularly in medical imaging, are often criticized for being "black boxes." While architectures like the UNet achieve high accuracy in tasks such as brain tumor segmentation, they lack the ability to explain the reasoning behind their predictions. This lack of interpretability remains a significant barrier to clinical adoption. Existing methods, such as saliency maps, often produce visualizations that are difficult for human experts to decode. This paper addresses the "symbol grounding problem" by introducing the Symbolic Semantic ($S^2$) framework, which links high-level neural representations to discrete, human-understandable symbols.

Methodology: The Symbolic UNet (SUNet)

The authors propose the Symbolic UNet (SUNet), a framework that forces a segmentation model to communicate image features through a private symbolic language. The architecture integrates a standard UNet with a communication loop consisting of two cooperative agents: a Sender and a Receiver.

The SUNet architecture combines a standard UNet with an Emergent Language module featuring Sender and Receiver LSTM agents.

The process begins with a base segmenter (UNet) extracting feature maps from the input image. These features are passed to the Sender (LSTM), which generates a "symbolic sentence"—a sequence of discrete integers from a predefined vocabulary. The Receiver (LSTM) then takes this sentence and attempts to reconstruct the segmentation mask. The final output is a combination of the original UNet features and the Receiver's reconstruction. To enable end-to-end training with discrete symbols, the authors employ the Gumbel-Softmax estimator. This setup forces the agents to evolve a "private language" that efficiently encodes critical image features.

Experimental Results

The model was evaluated using Fluid-Attenuated Inversion Recovery (FLAIR) brain MRIs from the The Cancer Imaging Archive (TCGA) dataset, involving 110 subjects with lower-grade gliomas.

Segmentation Performance

Surprisingly, the introduction of the symbolic communication layer improved segmentation accuracy. SUNet achieved a mean Dice Similarity Coefficient (DSC) of 90.0, a significant improvement over the baseline UNet's 81.4 DSC. This suggests that the symbolic "bottleneck" acts as a regularizer, forcing the network to focus on essential features and ignore noise.

Emergent Semantics and Interpretability

The core contribution of this work is the interpretability provided by the emergent language. Statistical analysis revealed that specific symbol sequences correlate strongly with clinical characteristics such as tumor presence, area, and eccentricity.

Visual examples of MRI slices categorized by clinical features (presence, area, eccentricity) alongside their corresponding emergent symbol sequences.

As shown in the visual analysis, the model generates consistent symbol prefixes for different tissue types. For instance, sequences starting with 657, 653 typically indicate normal tissue, while sequences starting with 8584 signal the presence of a tumor. Regression analysis confirmed this predictive power, with symbols achieving an $R^2$ of 0.67 for tumor presence and 0.44 for tumor area. The hierarchical nature of the LSTM-generated sentences means that early symbols capture high-level features, while later symbols refine shape details.

Conclusion

The SUNet framework demonstrates that symbolic communication can enhance both the performance and interpretability of medical image segmentation models. By grounding neural representations in discrete symbols, the model provides a pathway toward "reading" a network's internal logic. Future work aims to map these emergent symbols to natural language descriptions, further bridging the gap between artificial intelligence and clinical expertise.

Summary updated

BibTeX
@inproceedings{santamariapang2020towards,
  author = {Alberto Santamaría-Pang and James Kubricht and Aritra Chowdhury and Chitresh Bhushan and Peter H. Tu},
  title = {{Towards Emergent Language Symbolic Semantic Segmentation and Model Interpretability}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2020},
  doi = {10.1007/978-3-030-59710-8\_32},
}
Figure