TL;DR
This paper tackles the challenge of reliably detecting T-lymphocytes in cancer tissue images using deep learning, especially when data is limited or comes from different sources. They adapted a RetinaNet model and found that transfer learning significantly improves performance across various tumor types, offering practical recommendations for building robust diagnostic tools.
What Problem Does This Paper Address?
Immuno-oncology treatments, which harness the body's immune system to fight cancer, show great promise, but their effectiveness varies widely among patients. A key factor influencing treatment success is the presence and location of T-lymphocytes within the tumor microenvironment. Accurately quantifying these cells is crucial for predicting treatment response. While deep learning offers a powerful way to automate this process, existing methods struggle with robustness when applied to data from different pathology labs, staining protocols, or even different types of tumors (domain shift). This paper aims to evaluate and improve the robustness of deep learning models for T-lymphocyte detection in immunohistochemistry (IHC) images, particularly in scenarios with limited training data and across different tumor indications.
Methods
The researchers adapted the RetinaNet architecture, a popular object detection model, for the task of identifying T-lymphocytes (marked by CD3+ staining) in IHC images. They treated this as a multi-class object detection problem, classifying cells into three categories: tumor cells, CD3+ T-lymphocytes, and "non-specified" stromal cells.
To address the problem of domain shift and limited data, they employed transfer learning. This involves training a model on a source domain (e.g., one type of cancer) and then fine-tuning it on a target domain (another type of cancer) with minimal additional data. They systematically evaluated the impact of:
- WSI Ablation Study: Training the model with an increasing number of whole slide images (WSIs) to understand how data quantity affects performance and robustness.
- Deployment on Target Domains: Testing a model trained on one tumor type (HNSCC) on other tumor types without any further training.
- Model Fine-tuning: Fine-tuning the source-domain model on a small amount of data from a target domain to see if performance could be recovered.
They also introduced a semi-automatic annotation pipeline using commercially available software, which significantly speeds up the labeling process compared to purely manual annotation. They validated the quality of these semi-automatic annotations by comparing them to manual annotations from pathologists and using various kappa statistics to quantify inter-annotator agreement.
Datasets
The study utilized a dataset of 92 human tumor samples from four different tumor indications:
- Head and Neck Squamous Cell Carcinoma (HNSCC)
- Non-Small Cell Lung Cancer (NSCLC)
- Triple-Negative Breast Cancer (TNBC)
- Gastric Cancer (GC)
These samples were stained for CD3 (a marker for T-lymphocytes) and digitized. The researchers created a benchmark dataset by selecting regions of interest (ROIs) from these WSIs. They also provide public access to these selected ROIs on Zenodo.
Key Insights and Ideas
- Semi-automatic annotation is viable: The study demonstrates that semi-automatic annotation pipelines can generate high-quality labels efficiently, with agreement comparable to human annotators, making them a practical choice for large-scale studies.
- Transfer learning is crucial for cross-tumor robustness: When applying a model trained on one cancer type to another, performance drops significantly. Fine-tuning with even a single slide from the target domain can largely recover this performance, highlighting the power of transfer learning.
- Data quantity matters, but with diminishing returns: Increasing the number of training WSIs generally improves model performance and robustness, but this improvement plateaus after a certain point (around 9 WSIs for tumor cell detection).
- CD3+ cell detection is relatively easier: Detecting CD3+ T-lymphocytes is less sensitive to the number of training images compared to differentiating tumor and non-specified cells, suggesting that the IHC staining itself provides a strong signal.
- Tumor margins are challenging: Differentiating tumor cells from non-specified cells is particularly difficult at tumor boundaries, a challenge that also affects human annotators.
- Statistical effects can influence metrics: The ratio of CD3+ to non-specified cells in a sample can disproportionately affect performance metrics, especially for CD3+ cell detection, suggesting that careful consideration of class balance is needed.
Key Results
- The adapted RetinaNet model achieved human-level performance on their test set, with an average precision (AP) of 0.74 in-domain (within the same tumor type) and 0.72-0.74 cross-domain (across different tumor types) after fine-tuning.
- Training with more WSIs (up to 22) led to increased robustness, indicated by a lower variance in performance across model repetitions.
- Fine-tuning with just one additional WSI from a target domain significantly improved performance and robustness for models trained with fewer source domain images.
- For tumor cell detection, models trained with a higher number of WSIs showed a 10-15% improvement in AP compared to models trained with fewer WSIs, especially when deployed on target domains.
- The study provides concrete recommendations: use semi-automatic pipelines for annotation, aim for at least ~500 CD3+ cell annotations for robust detection, and ~5000 annotations per class for differentiating tumor/non-specified cells.
Limitations and Future Work
- Limited Scope of Artifacts: The study primarily focused on pre-selected regions of interest (ROIs) and did not extensively evaluate performance on complete WSIs containing various artifacts (staining issues, tissue folds, necrotic areas). Preliminary experiments showed that these artifacts can indeed challenge the algorithm.
- Inter-lab Variability: While the study used data from a single lab for training and digitization, they acknowledge that samples from different labs or digitized with different scanners can introduce domain shifts that may require fine-tuning.
- Future Work: The authors suggest exploring alternative domain shifts (different labs, scanners, staining agents) and using unsupervised domain adaptation techniques. They also propose integrating T-lymphocyte detection into more complex pipelines that first handle artifacts and segment tumor areas. Active learning is also mentioned as a way to iteratively improve annotations and minimize overhead.
Summary updated