Self Supervised Lesion Recognition For Breast Ultrasound Diagnosis

Yuanfan Guo, Canqian Yang, Tiancheng Lin, Chunxiao Li, Rui Zhang, Yi Xu

TL;DR

This paper introduces a multi-task learning framework for breast ultrasound diagnosis that leverages a novel self-supervised "Lesion Recognition" (LR) task. By using contrastive learning to pull together features from different views of the same lesion while pushing apart features from different lesions, the model learns more robust, view-invariant representations, leading to significant gains in classification accuracy and diagnostic consistency.

What Problem Does This Paper Address?

Breast Ultrasound (BUS) is a primary tool for early cancer detection, but interpreting these images is challenging. A single ultrasound image is a 2D slice of a 3D lesion; consequently, radiologists typically examine at least two different views (e.g., transverse and longitudinal) to form a complete picture.

However, most current deep learning-based Computer-Aided Diagnosis (CAD) systems treat every image as an independent data point. This approach ignores the semantic relationship between different views of the same lesion. The authors argue that this "image-level" paradigm fails to capture the 3D nature of the pathology and leads to inconsistent predictions across different views of the same patient.

Methods

The authors propose a multi-task framework that combines a standard classification task with a self-supervised auxiliary task called Lesion Recognition (LR).

  1. Feature Extraction: A ResNet-50 backbone ($f_\theta$) extracts features from ultrasound images.
  2. Main Task (Classification): A fully-connected head ($g_\phi$) predicts whether a lesion is Benign or Malignant using standard cross-entropy loss.
  3. Auxiliary Task (Lesion Recognition): This is the core innovation. Using a contrastive loss function, the model is trained to:
  4. Attract: Pull together the feature embeddings of different views belonging to the same lesion.
  5. Repel: Push apart the embeddings of views belonging to different lesions.
Overview of the proposed multi-task framework showing how views of the same lesion are pulled together (attracted) while views of different lesions are pushed apart (repelled).

Unlike standard Instance Recognition (IR)—which compares an image to its own augmented versions—LR compares different physical views of the same object. This encourages the model to learn features that are invariant to the viewpoint but sensitive enough to distinguish between two different tumors.

Datasets

The study utilized an in-house breast ultrasound dataset containing:

  • 5,911 images in total.
  • 2,131 unique lesions (2,002 benign images and 3,909 malignant images).
  • Data is naturally organized so that images of the same lesion are grouped, requiring no additional manual annotation for the LR task.

The researchers evaluated their approach using 5-fold cross-validation.

Key Insights and Ideas

  • Lesion-Level vs. Image-Level: The authors found that standard Instance Recognition (IR) actually degraded performance (AUC dropped by 0.8%). This is because IR encourages the model to distinguish between different views of the same lesion, whereas the goal of diagnosis is to recognize the underlying lesion regardless of the view.
  • Reducing Inconsistency: A major benefit of LR is reducing the "inner lesion mis-classification rate" (MCR). By forcing the model to see different views as the same entity, it becomes much less likely to label one view as benign and another as malignant for the same patient.
  • Hard Negatives Matter: The ablation study showed that "hard negatives"—images of different lesions that belong to the same semantic class (e.g., two different malignant tumors)—are more useful for training than "easy negatives" from different classes. Discriminating between similar-looking but distinct lesions forces the model to capture finer-grained features.

Key Results

The proposed LR framework outperformed both the baseline (classification only) and the standard IR approach across all metrics:

  • AUC Improvement: Reached 96.2%, a 1.5% increase over the baseline.
  • Accuracy (ACC): Improved by 2.8% (89.1% vs. 86.3%).
  • Consistency: The MCR (mis-classification rate across views) dropped by 7.2% compared to the baseline.
  • Specificity & Precision: These saw significant jumps of 6.8% and 3.9% respectively, suggesting the model is much better at avoiding false positives.
Table 2. Ablation study on negative samples with LR employed on classification task(mean±std%). ‘LR(-SC)’ denotes remov- ing negative samples from same semantic class. ‘LR(-DC)’ denotes removing negative samples from different semantic class. ‘LR(-)’ denotes removing all negative samples.

Limitations and Future Work

The authors acknowledge a few areas for further exploration:

  • Dataset Specificity: The results are based on a single in-house dataset. Testing on diverse, multi-center public datasets would be necessary to prove generalizability.
  • Architecture: The study focused on ResNet-50; exploring how Vision Transformers (ViTs) or other architectures handle this multi-view contrastive task could be fruitful.
  • Integration: While the tasks share a backbone, the final classification is still done on individual images. Future work could explore explicit multi-view fusion layers that aggregate information from all available views before making a final diagnostic call.

Summary updated

Abstract

Previous deep learning based Computer Aided Diagnosis (CAD) system treats multiple views of the same lesion as independent images. Since an ultrasound image only describes a partial 2D projection of a 3D lesion, such paradigm ignores the semantic relationship between different views of a lesion, which is inconsistent with the traditional diagnosis where sonographers analyze a lesion from at least two views. In this paper, we propose a multi-task framework that complements Benign/Malignant classification task with lesion recognition (LR) which helps leveraging relationship among multiple views of a single lesion to learn a complete representation of the lesion. To be specific, LR task employs contrastive learning to encourage representation that pulls multiple views of the same lesion and repels those of different lesions. The task therefore facilitates a representation that is not only invariant to the view change of the lesion, but also capturing fine-grained features to distinguish between different lesions. Experiments show that the proposed multi-task framework boosts the performance of Benign/Malignant classification as two sub-tasks complement each other and enhance the learned representation of ultrasound images.
BibTeX
@misc{guo2022self,
  author = {Yuanfan Guo and Canqian Yang and Tiancheng Lin and Chunxiao Li and Rui Zhang and Yi Xu},
  title = {{Self Supervised Lesion Recognition For Breast Ultrasound Diagnosis}},
  year = {2022},
  eprint = {2204.08477},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2204.08477},
}
Figure