TASL-Net: Tri-attention selective learning network for intelligent diagnosis of bimodal ultrasound video

Chengqian Zhao, Zhao Yao, Zhaoyu Hu, Yuanxin Xie, Yafang Zhang, Yuanyuan Wang, Shuo Li, Jianhua Zhou, Jianqiao Zhou, Yin Wang, Jinhua Yu

TL;DR

TASL-Net is a novel deep learning framework that mimics the diagnostic intuition of sonographers by embedding "tri-attention" (temporal, spatial, and bimodal) into a mutual transformer network. By focusing on clinically relevant perfusion patterns and structural features, it achieves state-of-the-art performance in diagnosing lung, breast, and liver cancers from bimodal ultrasound videos.

What Problem Does This Paper Address?

Medical diagnosis using bimodal ultrasound—which combines traditional gray-scale (GSUS) for anatomy and contrast-enhanced (CEUS) for blood flow—is a gold standard for many cancers. However, for deep learning models, these videos present a "black box" challenge: they are long, contain massive amounts of redundant data, and require a nuanced understanding of how blood perfusion changes over time.

Human sonographers don't watch the whole video with equal intensity; they have "diagnostic attention." They look for the exact moment contrast enters a lesion (temporal), they focus on the differences between the tumor and surrounding tissue (spatial), and they constantly correlate the grainy structural map of GSUS with the bright perfusion map of CEUS (bimodal). Existing AI models often struggle to integrate this domain knowledge, leading to high computational costs and lower clinical reliability.

Methods

The authors introduce TASL-Net (Tri-Attention Selective Learning Network), which breaks down the sonographer’s expertise into three core modules:

  1. TIC-based Video Selector (Temporal Attention): Instead of processing the entire video, this module uses Time-Intensity Curves (TIC) and a Savitzky-Golay filter to automatically identify and "clip" the moments of rapid perfusion. This removes "junk" frames and drastically improves efficiency.
  2. Earliest-Enhanced TIC Analysis (ETIC) (Spatial Attention): This module uses Structural Similarity (SSIM) variation to detect exactly where the contrast agent arrives first. It then uses a pre-trained MobileNetV2 and K-means clustering to differentiate between the lesion, healthy tissue, and the thoracic wall, ensuring the model compares the right areas.
  3. Mutual Convolutional Transformer (MCT) (Bimodal Attention): This is the "brain" of the system. It uses a hybrid architecture where Convolutional Neural Networks (CNNs) capture local textures and edges (best for GSUS), while Transformers model long-term temporal dependencies (best for CEUS). A "feature sharing" unit allows these two branches to communicate, bridging the semantic gap between the two imaging modes.

The model is optimized using a Maximum Mean Discrepancy (MMD) loss, which aligns the quantified intensity data from the TICs with the visual features extracted from the video frames.

Datasets

The study validated TASL-Net on three distinct clinical datasets, totaling 791 cases:

  • Lung Dataset: 563 bimodal videos (Shanghai Pulmonary Hospital).
  • Breast Dataset: 108 bimodal videos (Ruijin Hospital).
  • Liver Dataset: 120 bimodal videos (Sun-Yat Sen University Cancer Center).

Notably, the authors did not use "clean" data; they kept the noise and variations typical of real-world clinical settings to ensure the model's robustness.

Key Insights and Ideas

  • Domain Knowledge as an Architecture: The biggest "aha moment" is that better performance doesn't just come from deeper networks, but from structured networks. By forcing the AI to follow the same workflow as a human expert (Select clip -> Identify key spots -> Compare modes), the model becomes more accurate and interpretable.
  • SSIM for Perfusion Detection: Using SSIM—a metric usually used for image quality—to detect the arrival of contrast agents is an innovative way to treat functional changes as structural variations.
  • Efficiency through Selection: By filtering for "rapidly rising" perfusion moments, the authors solved the memory bottleneck that usually plagues high-resolution medical video analysis.
  • Bimodal Synergy: The results prove that GSUS and CEUS are truly complementary; GSUS provides the "where" (anatomy), while CEUS provides the "what" (malignancy via blood flow).

Key Results

TASL-Net outperformed five state-of-the-art (SOTA) methods, including TimeSformer and Video Swin Transformer (VST):

  • Diagnostic Accuracy: Achieved an average accuracy improvement of 6.43% over the next best method.
  • AUC Scores: Reached 0.86 for Lung, 0.86 for Breast, and an impressive 0.97 for Liver datasets.
  • Ablation Success: The authors proved that removing any of the three "attentions" led to a significant drop in performance, confirming that the "tri-attention" approach is the primary driver of its success.

Limitations and Future Work

While TASL-Net is a significant step forward, the authors acknowledge a few hurdles:

  • Dataset Diversity: While they tested on three organs, the data still comes from specific high-level medical centers. Future work needs to include more diverse samples from various hospitals to ensure global generalizability.
  • Computational Cost: Even with the video selector, bimodal video analysis is resource-intensive. The team aims to develop more "lightweight" versions of the network for use in point-of-care devices.
  • Foundation Models: The authors expressed interest in exploring how their tri-attention strategy could be integrated with emerging "large vision models" in the future.

Summary updated

BibTeX
@article{zhao2025tasl,
  author = {Chengqian Zhao and Zhao Yao and Zhaoyu Hu and Yuanxin Xie and Yafang Zhang and Yuanyuan Wang and Shuo Li and Jianhua Zhou and Jianqiao Zhou and Yin Wang and Jinhua Yu},
  title = {{TASL-Net: Tri-attention selective learning network for intelligent diagnosis of bimodal ultrasound video}},
  journal = {Expert Systems with Applications},
  year = {2025},
  doi = {10.1016/j.eswa.2025.128355},
}
Figure