Deep learning models in medical imaging are often criticized for their "black box" nature. While accurate, they lack the ability to explain the reasoning behind a classification. Researchers at GE Research have introduced ESCELL (Emergent Symbolic Cellular Language), a framework that addresses this by teaching AI agents to communicate using a discrete symbolic language to identify cell types.
The Interpretability Gap in Healthcare
Standard deep learning relies on high-dimensional feature spaces that are opaque to human observers. In clinical settings, transparency is vital; a pathologist must understand why a model identifies a cell as cancerous. ESCELL bridges this gap by forcing agents to use a discrete vocabulary of symbols. If an agent consistently uses "Symbol 62" for a T-cell, it creates a human-readable link between raw data and biological phenotypes.
Methodology: The Lewis Signaling Game
The core of ESCELL is a Lewis’ signaling game, a framework from linguistics and game theory. The setup involves two agents:
- The Sender: Analyzes a target cell and selects a single symbol from a fixed vocabulary (size 100).
- The Receiver: Receives the symbol and a set of images (the target plus distractors). It must use the symbol to identify the correct target.
To enable backpropagation with discrete symbols, the researchers employed the Gumbel Softmax trick, allowing the model to learn categorical distributions while remaining differentiable.
Experimental Setup and Datasets
The study utilized a dataset of 4,125 immune cells from a colon cohort, imaged via the Cell DIVE™ hyperplex immunofluorescence platform. Cells were stained for four markers: CD3 (T-cells), CD20 (B-cells), CD68 (Macrophages), and Claudin1 (Epithelial cells), plus a negative control. Instead of raw pixels, the model processed 28 extracted features (shape and intensity statistics) per cell.
Two experimental scenarios were tested:
- Experiment 1: The Sender sees both the target and distractors.
- Experiment 2: The Sender only sees the target image, requiring a more robust description.
Key Results and Visual Validation
The agents demonstrated high identification accuracy: 93.18% in Experiment 1 and 78.78% in the more challenging Experiment 2. Analysis of symbol distributions showed clear clustering, where specific symbols naturally became "codes" for specific cell types (e.g., Symbol 62 for CD3-positive cells).
Visual validation confirmed that these symbols were consistently assigned to the same cell types across entire tissue samples. While some ambiguity remained—such as using the same symbol for cells with very weak staining—the emergent language effectively reflected the inherent biological nuances of the data.
Future Directions
Current limitations include the reliance on pre-extracted features. Future iterations aim to use Convolutional Neural Networks (CNNs) to generate symbols directly from raw pixels. Additionally, "grounding" these symbols—mapping "Symbol 62" to the natural language term "T-cell"—will be the final step in creating a fully interpretable diagnostic tool.
Summary updated