Escell: Emergent Symbolic Cellular Language

Aritra Chowdhury, James R. Kubricht, Anup Sood, Peter H. Tu, Alberto Santamaría-Pang

Deep learning models in medical imaging are often criticized for their "black box" nature. While accurate, they lack the ability to explain the reasoning behind a classification. Researchers at GE Research have introduced ESCELL (Emergent Symbolic Cellular Language), a framework that addresses this by teaching AI agents to communicate using a discrete symbolic language to identify cell types.

The Interpretability Gap in Healthcare

Standard deep learning relies on high-dimensional feature spaces that are opaque to human observers. In clinical settings, transparency is vital; a pathologist must understand why a model identifies a cell as cancerous. ESCELL bridges this gap by forcing agents to use a discrete vocabulary of symbols. If an agent consistently uses "Symbol 62" for a T-cell, it creates a human-readable link between raw data and biological phenotypes.

Methodology: The Lewis Signaling Game

The core of ESCELL is a Lewis’ signaling game, a framework from linguistics and game theory. The setup involves two agents:

  1. The Sender: Analyzes a target cell and selects a single symbol from a fixed vocabulary (size 100).
  2. The Receiver: Receives the symbol and a set of images (the target plus distractors). It must use the symbol to identify the correct target.
The Sender-Receiver architecture used in the Lewis signaling game.

To enable backpropagation with discrete symbols, the researchers employed the Gumbel Softmax trick, allowing the model to learn categorical distributions while remaining differentiable.

Experimental Setup and Datasets

The study utilized a dataset of 4,125 immune cells from a colon cohort, imaged via the Cell DIVE™ hyperplex immunofluorescence platform. Cells were stained for four markers: CD3 (T-cells), CD20 (B-cells), CD68 (Macrophages), and Claudin1 (Epithelial cells), plus a negative control. Instead of raw pixels, the model processed 28 extracted features (shape and intensity statistics) per cell.

Two experimental scenarios were tested:

  • Experiment 1: The Sender sees both the target and distractors.
  • Experiment 2: The Sender only sees the target image, requiring a more robust description.

Key Results and Visual Validation

The agents demonstrated high identification accuracy: 93.18% in Experiment 1 and 78.78% in the more challenging Experiment 2. Analysis of symbol distributions showed clear clustering, where specific symbols naturally became "codes" for specific cell types (e.g., Symbol 62 for CD3-positive cells).

Emergent symbols overlaid on a tissue image, demonstrating consistent cell type identification.

Visual validation confirmed that these symbols were consistently assigned to the same cell types across entire tissue samples. While some ambiguity remained—such as using the same symbol for cells with very weak staining—the emergent language effectively reflected the inherent biological nuances of the data.

Future Directions

Current limitations include the reliance on pre-extracted features. Future iterations aim to use Convolutional Neural Networks (CNNs) to generate symbols directly from raw pixels. Additionally, "grounding" these symbols—mapping "Symbol 62" to the natural language term "T-cell"—will be the final step in creating a fully interpretable diagnostic tool.

Summary updated

BibTeX
@inproceedings{chowdhury2020escell,
  author = {Aritra Chowdhury and James R. Kubricht and Anup Sood and Peter H. Tu and Alberto Santamaría-Pang},
  title = {{Escell: Emergent Symbolic Cellular Language}},
  booktitle = {ISBI},
  year = {2020},
  doi = {10.1109/isbi45749.2020.9098343},
}
Figure