Abstract
This study investigates the effects of radio-opaque artefacts, such as skin markers, breast implants, and pacemakers, on mammography classification models. After manually annotating 22,012 mammograms from the publicly available EMBED dataset, a robust multi-label artefact detector was developed to identify five distinct artefact types (circular and triangular skin markers, breast implants, support devices and spot compression structures). Subsequent experiments on two clinically relevant tasks — breast density assessment and cancer screening — revealed that these artefacts can significantly affect model performance, alter classification thresholds, and distort output distributions. These findings underscore the importance of accurate automatic artefact detection for developing reliable and robust classification models in digital mammography. To facilitate future research, our annotations, code, and model predictions are made publicly available.
What Problem Does This Paper Address?
Deep learning models have achieved remarkable success in breast cancer screening, often matching or exceeding expert performance. However, these models are frequently "black boxes" that can be sensitive to subtle image characteristics. While research has explored how different X-ray hardware affects model performance, the impact of radio-opaque artefacts—localized, high-contrast objects like skin markers, implants, and medical devices—has been largely overlooked.
The core problem is twofold:
- Lack of Annotated Data: Most public mammography datasets lack labels for these artefacts, hindering systematic study of their effects.
- Model Bias and Shortcuts: If a model learns that a specific artefact (like a triangular marker placed near a palpable lump) is a shortcut for a "cancer" diagnosis, it fails to learn the underlying biological features of malignancy. This leads to poor generalization and potential diagnostic errors in clinical settings.
Methods
The researchers addressed these challenges in three phases:
- Annotation & Dataset Creation: They manually labeled 22,012 mammograms from the EMory BrEast imaging Dataset (EMBED) for five artefact types: circular markers, triangular markers, breast implants, support devices (e.g., pacemakers), and spot compression structures. This dataset is notably diverse and contains approximately 3.4 million images.
- Artefact Detection Model: A multi-label ResNet-50 classifier was trained to automatically detect these five artefacts. The model employs a shared encoder with five separate binary classification heads, enabling the identification of multiple artefacts within a single image.
- Downstream Impact Analysis: To quantify the impact on clinical tasks, the authors trained two ResNet-18 models: one for binary breast cancer screening and one for four-class breast density assessment. They then compared the performance of these models on images with and without artefacts.
Key Insights and Ideas
- The "Shortcut" Effect: Triangular markers are particularly problematic. Because radiologists often place them near palpable masses, AI models can learn to associate the shape of the marker itself with a positive cancer finding. This leads to a significant drop in specificity, where the model flags "cancer" primarily due to the presence of the marker.
- Distribution Distortion: Artefacts cause a systematic shift in the model’s output distribution. A classification threshold optimized for "clean" images may be entirely inappropriate for images containing implants or markers, leading to miscalibration.
- High Prevalence: Approximately 22% of the images in the EMBED dataset contain at least one artefact. This high prevalence suggests that if models are not trained to handle these features, nearly a quarter of clinical predictions could be compromised.
Key Results
- Artefact Detector Performance: The detector achieved an exceptional average ROC-AUC of .993 across all artefact types, demonstrating that automatic detection is highly feasible.
- Cancer Screening Degradation: While the screening model achieved an ROC-AUC of 0.86 on images without artefacts, performance dropped significantly on images with triangular markers (0.79) and breast implants (0.72).
- Density Assessment Impact: For breast density classification, balanced accuracy fell from 0.80 (no artefacts) to 0.67 for images containing implants.
- Calibration Issues: Images with triangular markers were almost systematically predicted as "positive" for cancer, regardless of the actual tissue state, highlighting a severe lack of model robustness to clinical markers.
Limitations and Future Work
This study focuses on identifying and quantifying the problem. While it proves that artefacts cause performance degradation, it does not evaluate specific mitigation strategies. Future research could explore data augmentation (digitally inserting artefacts during training) or adversarial training to make models invariant to these objects. The release of these labels allows for the development of "artefact-aware" models that can dynamically adjust confidence thresholds or ignore these features to ensure more reliable diagnostic support.
Summary updated