Real-Time Automatic M-Mode Echocardiography Measurement With Panel Attention

Ching-Hsun Tseng, S. C. Chien, Po-Shen Wang, Shin‐Jye Lee, Bin Pu, Xiao‐Jun ZengIEEE JBHI 2024 · DOI

TL;DR

This paper introduces RAMEM, a novel system for real-time automatic measurement of M-mode echocardiograms. It addresses key challenges in cardiac diagnostics by proposing a new "panel attention" mechanism for better object detection, a new dataset (MEIS) for M-mode echocardiography, and an efficient measurement algorithm (AMEM). This work aims to improve accuracy, reduce diagnosis time, and overcome the limitations of manual analysis.

What Problem Does This Paper Address?

Diagnosing heart conditions using echocardiography, particularly M-mode imaging, is crucial but faces several hurdles. Manual analysis is time-consuming and prone to inter-observer variability, leading to inconsistent results. Existing deep learning approaches struggle with the unique characteristics of echocardiograms, such as large objects occupying significant pixel space and the need to capture long temporal sequences (from systole to diastole). Current CNN backbones have limited receptive fields, making it hard to grasp these extended cardiac cycles. While non-local attention mechanisms can capture global context, they often come with high computational overhead or lose critical information. Furthermore, there's a lack of publicly available datasets specifically for M-mode echocardiography instance segmentation, hindering the development of robust automated systems.

Methods

The authors propose RAMEM, a comprehensive framework built upon the YOLACT real-time instance segmentation (RIS) model. Key innovations include:

  • Panel Attention: This novel attention mechanism is designed to address the limited receptive field of CNNs and the computational burden of traditional non-local attention. It combines local and global attention by using pixel-unshuffling for down-sampling, allowing for a wider receptive field without significant information loss or excessive computation. This is embedded within an updated UPANets V2 backbone.
  • AMEM (Automatic M-mode Echocardiography Measurement): This algorithm acts as a post-processing step after instance segmentation. It leverages medical expertise to automatically extract key cardiac indicators from the segmented masks, focusing on specific measurements for the Aortic Valve (AV) and Left Ventricle (LV).
  • UPANets V2 Backbone: The authors utilize and update the UPANets V2 architecture as their backbone, integrating the panel attention mechanism to enhance its ability to handle large objects and global context.

Datasets

  • MEIS (M-mode Echocardiograms for Instance Segmentation): This is a newly introduced dataset specifically for M-mode echocardiography instance segmentation. It comprises 2,639 images (1024x768 resolution) with annotations for four object classes: Aortic Root (AoR), Left Atrium (LA), Interventricular Septum (IVS), and Left Ventricular Posterior Wall (LVPW). The dataset is split into training (1,521 images) and testing (1,118 images) sets.
  • PASCAL 2012 SBD: This standard dataset was used for evaluating the effectiveness of the panel attention mechanism and comparing different backbone configurations in a general object detection context.

Key Insights and Ideas

  • The "Big Object" Problem: Echocardiograms often feature large structures that span significant portions of the image. Standard CNNs struggle with this due to their limited receptive fields, making it difficult to capture the full context of cardiac motion.
  • Bridging Local and Global Attention: The panel attention mechanism is a clever way to achieve both local feature extraction (like CNNs) and global context awareness (like non-local attention) efficiently, particularly during down-sampling.
  • The Need for Domain-Specific Data: The introduction of the MEIS dataset is crucial, as existing general-purpose datasets may not adequately represent the unique challenges of M-mode echocardiography.
  • Manual vs. Automatic Measurement: The paper highlights that while manual measurements are the current standard, they are prone to bias. Automated methods, when properly designed, can achieve comparable or superior accuracy with greater consistency.

Key Results

  • Performance on PASCAL 2012 SBD: RAMEM, particularly with the UPANet80 V2 backbone and panel attention, achieved competitive results on the PASCAL 2012 SBD dataset, outperforming other RIS backbones and demonstrating the efficacy of the proposed attention mechanism.
  • Performance on MEIS: On the newly introduced MEIS dataset, RAMEM (using UPANet80 V2) significantly surpassed human performance and other methods like MENN in terms of accuracy (lower MAE and MSE) and speed (higher FPS). For example, the mean MAE for LV measurements was 0.425 for humans and 0.206 for AMEM, with AMEM achieving an FPS of ~0.041 seconds.
  • Real-time Capability: The proposed AMEM algorithm, when integrated with the UPANet80 V2 backbone, achieved real-time performance, with an FPS of 24, enabling seamless diagnosis.

Limitations and Future Work

  • MEIS Dataset Size: The authors acknowledge that the MEIS dataset, while the first of its kind for M-mode echocardiography instance segmentation, is relatively small compared to general object detection datasets. They suggest that a larger dataset with more labeled images and potentially video formats would be beneficial.
  • Panel Attention Complexity: While panel attention reduces complexity compared to traditional non-local methods, it is still more computationally intensive than pure CNNs.
  • AMEM Time Cost and Mask Reliance: The AMEM algorithm, despite its efficiency, still contributes to the overall computational overhead. Its accuracy is also highly dependent on the quality of the predicted masks. Defective masks can lead to inaccurate measurements, particularly for structures like the IVS in diastole.
  • Future Work: The authors suggest exploring ways to boost pixel searching for AMEM, potentially through regression prediction heads and GPU acceleration, to further reduce time costs. They also propose investigating non-parametric attention methods to avoid hyperparameter tuning issues and potentially mitigate the dilated effect seen in pixel-unshuffling. Combining AMEM with MENN's approach of multiple point selections could also balance accuracy and FPS.

Summary updated

Abstract

Motion mode (M-mode) echocardiography is essential for measuring cardiac dimension and ejection fraction. However, the current diagnosis is time-consuming and suffers from diagnosis accuracy variance. This work resorts to building an automatic scheme through well-designed and well-trained deep learning to conquer the situation. That is, we proposed RAMEM, an automatic scheme of real-time M-mode echocardiography, which contributes three aspects to address the challenges: 1) provide MEIS, the first dataset of M-mode echocardiograms, to enable consistent results and support developing an automatic scheme; For detecting objects accurately in echocardiograms, it requires big receptive field for covering long-range diastole to systole cycle. However, the limited receptive field in the typical backbone of convolutional neural networks (CNN) and the losing information risk in non-local block (NL) equipped CNN risk the accuracy requirement. Therefore, we 2) propose panel attention embedding with updated UPANets V2, a convolutional backbone network, in a real-time instance segmentation (RIS) scheme for boosting big object detection performance; 3) introduce AMEM, an efficient algorithm of automatic M-mode echocardiography measurement, for automatic diagnosis; The experimental results show that RAMEM surpasses existing RIS schemes (CNNs with NL & Transformers as the backbone) in PASCAL 2012 SBD and human performances in MEIS.
BibTeX
@article{tseng2024real,
  author = {Ching-Hsun Tseng and S. C. Chien and Po-Shen Wang and Shin‐Jye Lee and Bin Pu and Xiao‐Jun Zeng},
  title = {{Real-Time Automatic M-Mode Echocardiography Measurement With Panel Attention}},
  journal = {IEEE Journal of Biomedical and Health Informatics},
  year = {2024},
  doi = {10.1109/jbhi.2024.3413628},
}
Figure