TL;DR
This paper reveals a significant, often overlooked problem: inter-rater bias in medical image annotations can directly influence and bias the training of deep neural networks (DNNs) for segmentation. This bias leads to DNNs that perform differently on the same input data depending on which rater's annotations were used for training, with potentially worrisome clinical implications for lesion load and volume estimations.
What Problem Does This Paper Address?
Manual annotation of medical images is a cornerstone of supervised machine learning, but it's inherently subjective. Different human experts (raters) can delineate the same anatomical structures or pathologies with varying degrees of accuracy and consistency, a phenomenon known as inter-rater variability. While this variability has been acknowledged, the prevailing assumption is that objective, model-based segmentation approaches, and especially data-driven deep neural networks (DNNs), can overcome this by learning a "true" representation. This paper challenges that assumption, focusing specifically on inter-rater bias – systematic differences between raters, not just random errors. The authors argue that this bias, when used to train DNNs, doesn't disappear but is instead amplified and embedded within the model, leading to biased segmentation outputs. This is particularly concerning as DNNs are increasingly used for critical clinical tasks like disease progression monitoring and surgical planning.
Methods
The core of the methodology involves training 3D U-Net architectures for semantic segmentation. The key experimental design revolves around training identical U-Net models twice for each dataset: once using annotations from rater #1 and again using annotations from rater #2. The performance of these models is then evaluated through cross-evaluation (training on one rater's data, testing on the other's) and by comparing their outputs to the original manual annotations.
To further investigate the rater bias, a separate classifier DNN was trained. This classifier was tasked with distinguishing between the segmentation outputs of the two raters, and crucially, also between the segmentation predictions of the two U-Nets trained on different raters' data. This classifier DNN was used to assess whether the bias introduced by rater differences was detectable and if it was amplified in the DNN's outputs.
The paper also explores a "mixed source training" approach, simulating a scenario where one rater is more experienced than the other, and investigating how incorporating a small portion of expert annotations can influence the DNN's performance.
Datasets
The study utilizes two publicly available datasets:
- ISBI 2015 Multiple Sclerosis (MS) challenge dataset: This dataset consists of multi-modal brain MRI scans with annotations for MS lesions. It's notable for having annotations from two raters with different levels of expertise (4 years vs. 10 years of experience).
- Intracerebral Hemorrhage (ICH) CT scans: This dataset comprises CT scans of ICH patients, with annotations provided by an experienced radiologist (manual) and with the aid of a semi-manual segmentation tool.
The data was partitioned into overlapping patches for training due to GPU memory limitations.
Key Insights and Ideas
- Bias Amplification: The most striking finding is that inter-rater bias isn't just preserved but actively amplified by DNN training. A DNN trained on one rater's data produces segmentations that are systematically different from those produced by a DNN trained on another rater's data, even when using the same network architecture and training regime.
- "Ground Truth" is Subjective: The study highlights that the "ground truth" used to train DNNs is not an objective reality but a reflection of the annotator's subjective interpretation. This subjective interpretation becomes embedded in the DNN.
- Classifier Detects Amplified Bias: A classifier DNN trained to distinguish between raters performed better when classifying the outputs of the segmentation DNNs than when classifying the original manual annotations. This strongly suggests that the bias is not only present but also more pronounced and consistent in the DNN's predictions.
- Clinical Implications are Real: The paper demonstrates concrete clinical consequences. For MS lesion load estimation, a DNN trained on the less experienced rater's data consistently underestimates the load. Similarly, ICH volume estimates show consistent and larger discrepancies when derived from DNNs trained on different sources compared to the differences between the original manual and semi-manual annotations.
- Mixed Training Mitigates Bias: Introducing even a small proportion of annotations from a more experienced rater into the training set can significantly influence the DNN's output, mitigating the bias introduced by a less experienced rater.
Key Results
- Cross-Rater Dice Scores: When a DNN trained on rater #1's annotations was tested on data annotated by rater #2 (and vice-versa), the Dice scores were significantly lower than when trained and tested on the same rater's data. For the MS dataset, this difference was statistically significant (p-value of 9.9029e-04). For the ICH dataset, it was 0.0167.
- Classifier Performance:
- For the MS dataset, a classifier DNN achieved hit rates of 0.87 (source vs. source) and 0.917 (source vs. source) for manual annotations. When tested on DNN predictions, these hit rates increased to 0.90 (network vs. network) and 0.93 (network vs. network).
- For the ICH dataset, the hit rates were 0.90 (source vs. source) and 0.917 (source vs. source) for manual annotations, and 0.93 (network vs. network) and 0.95 (network vs. network) for DNN predictions.
- MS Lesion Load Underestimation: The study showed that network #1 (trained on the less experienced rater) consistently estimated lower MS lesion loads than network #2 (trained on the more experienced rater), with this difference being much larger than the difference between the raters' own estimates.
- ICH Volume Differences: Similar to MS lesions, ICH volumes calculated from network #1 were consistently higher than from network #2, and these differences were larger and more consistent than those between the original manual and semi-manual annotations.
Limitations and Future Work
- Dataset Specificity: The authors acknowledge that their findings are dataset-specific, but they believe the phenomenon of DNN-induced bias is general and warrants community awareness.
- Mixed Training Strategy: While demonstrating the effectiveness of mixed training, the paper suggests further exploration into optimal strategies for combining annotations from multiple sources, especially when dealing with varying levels of expertise.
- Clinical Validation: The study highlights worrisome clinical implications, but further direct clinical validation would be necessary to fully quantify the impact of this bias on patient outcomes.
- Annotator Expertise: The paper strongly suggests that the expertise of the annotator directly influences DNN training. Future work could focus on developing methods to quantify annotator expertise and incorporate this information into the training process.
Summary updated