Boosting Breast Ultrasound Video Classification by the Guidance of Keyframe Feature Centers

AnLan Sun, Zhao Zhang, Meng Lei, Yuting Dai, Dong Wang, Liwei Wang

TL;DR

This paper introduces KGA-Net, a novel deep learning model that significantly improves breast ultrasound video classification by leveraging both video and static image datasets. It addresses the challenge of limited video data by using static images to guide frame attention in video analysis, leading to more accurate and interpretable diagnoses.

What Problem Does This Paper Address?

Breast cancer is a major health concern, and ultrasound is a primary diagnostic tool. While ultrasound videos offer richer information than static images, classifying them accurately is challenging due to the difficulty in collecting large, annotated video datasets. Sonographers often capture static images rather than full videos, and prospectively collected videos require extensive effort to link with pathological outcomes. This scarcity of video data limits the performance of deep learning models. Existing methods often rely on keyframe supervision, which is not always available. This paper aims to bridge this gap by effectively utilizing readily available static image datasets to enhance the performance of breast ultrasound video classification.

Methods

The core of the proposed method is the Keyframe Guided Attention Network (KGA-Net). This network takes both video clips and static images as input during training.

  • Dual Input Training: KGA-Net utilizes a shared 2D CNN backbone for both video frame feature extraction and static image feature extraction. This allows the model to learn from both modalities simultaneously.
  • Frame Attention Module: For video classification, an attention module assigns weights to each frame, indicating its importance for the final diagnosis. These weighted features are then aggregated to form a video representation.
  • Coherence Loss: This is the novel contribution. It uses feature centers (mean feature vectors for benign and malignant classes) derived from the larger static image dataset to guide the frame attention mechanism. The coherence loss encourages frames that are closer to their respective class centers (i.e., more discriminative) to receive higher attention weights, while frames further away get lower weights. This effectively directs the model's focus to the most informative frames.
  • Total Loss: The overall training objective combines cross-entropy loss for video and image classification, center loss for feature center learning, and the proposed coherence loss.

Datasets

  • BUSV Dataset: Used for video classification. It contains 113 malignant and 75 benign breast ultrasound videos. The paper uses the official data split provided in the original BUSV paper.
  • BUSI Dataset: Used as the static image dataset. It comprises 445 benign and 210 malignant breast ultrasound images. This dataset is used to train the feature centers that guide the video model.

Key Insights and Ideas

  • Leveraging Static Images for Video: The paper demonstrates that static image datasets, even if not directly video data, can be highly effective in guiding video analysis by providing robust feature centers.
  • Feature Centers as Diagnostic Guides: The concept of using class-specific feature centers derived from a larger dataset to inform attention mechanisms in a different modality (video) is a powerful idea for data-scarce scenarios.
  • Coherence Loss for Frame Importance: The coherence loss effectively translates the "quality" of a frame (its proximity to a class center) into an attention weight, making the model focus on diagnostically relevant frames.
  • Interpretability through Keyframes: The attention weights naturally highlight important frames, offering a form of interpretability that aligns with how clinicians might review ultrasound videos.

Key Results

  • Significant Performance Boost: KGA-Net achieved state-of-the-art results on the BUSV dataset, outperforming traditional video classification models (I3D, SlowFast, R(2+1)D, CSN) and a transformer-based model (MVIT) by a considerable margin.
  • Quantitative Metrics: On the BUSV test set, KGA-Net achieved an AUC of 94.67%, ACC of 89.74%, Sensitivity of 88.46%, and Specificity of 92.31%.
  • Ablation Study: Removing image guidance or the coherence loss module led to a significant drop in performance, highlighting their crucial contributions. The coherence loss was particularly important for improving sensitivity.
  • Visual Analysis: The paper presents visualizations showing that frames with higher attention weights indeed exhibit clearer diagnostic features, supporting the method's interpretability. The relationship between attention weights and feature distances was shown to be linear, validating the coherence loss's effectiveness.

Limitations and Future Work

  • Dataset Size: While the paper effectively uses a static image dataset to augment video data, the BUSV dataset itself is relatively small (188 videos).
  • Generalizability: The performance of the feature centers is dependent on the quality and representativeness of the static image dataset.
  • Future Work: The authors suggest that the attention weights provide a new perspective for keyframe extraction tasks in ultrasound videos, implying that this method could be extended for automated keyframe selection. Further exploration of different backbone architectures and attention mechanisms could also be beneficial.

Summary updated

Abstract

Breast ultrasound videos contain richer information than ultrasound images, therefore it is more meaningful to develop video models for this diagnosis task. However, the collection of ultrasound video datasets is much harder. In this paper, we explore the feasibility of enhancing the performance of ultrasound video classification using the static image dataset. To this end, we propose KGA-Net and coherence loss. The KGA-Net adopts both video clips and static images to train the network. The coherence loss uses the feature centers generated by the static images to guide the frame attention in the video model. Our KGA-Net boosts the performance on the public BUSV dataset by a large margin. The visualization results of frame attention prove the explainability of our method. The codes and model weights of our method will be made publicly available.
BibTeX
@inproceedings{sun2023boosting,
  author = {AnLan Sun and Zhao Zhang and Meng Lei and Yuting Dai and Dong Wang and Liwei Wang},
  title = {{Boosting Breast Ultrasound Video Classification by the Guidance of Keyframe Feature Centers}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2023},
  doi = {10.1007/978-3-031-43904-9\_43},
}
Figure