Robustness of electrocardiogram signal quality indices

Saifur Rahman, Chandan Karmakar, Iynkaran Natgunanathan, John Yearwood, Marimuthu PalaniswamiJournal of The Royal Society Interface 2022 · DOI

TL;DR

This paper investigates the robustness of common statistical signal quality indices (SSQIs) for electrocardiogram (ECG) signals. It finds that while these indices can achieve high accuracy, their performance is highly sensitive to the dataset used, and they suffer from low specificity, making them unreliable for clinical decision-making without further improvements. The authors suggest adaptive thresholds or machine learning approaches to overcome these limitations.

What Problem Does This Paper Address?

Wearable health monitoring devices are increasingly popular, but their reliability is hampered by the presence of noise in the physiological signals they capture, particularly ECG. This noise can lead to incorrect diagnoses or false alarms. To address this, signal quality indices (SQIs) are used to identify and flag noisy ECG segments before they are processed by diagnostic systems. However, existing statistical SQIs (SSQIs) often rely on fixed thresholds and have shown inconsistent performance across different datasets and noise conditions. This paper aims to systematically evaluate the robustness of these SSQIs, specifically examining how their performance is affected by dataset diversity and the choice of analysis window size.

Methods

The authors evaluated five popular statistical signal quality indices (SSQIs):

  • Kurtosis (SQI<sub>kur</sub>): Measures the "tailedness" of the signal's distribution.
  • Relative Power of QRS Complex (SQI<sub>p</sub>): Compares the power spectral density of the QRS complex to the overall ECG signal.
  • Signal-to-Noise Ratio (SQI<sub>snr</sub>): A direct measure of signal strength relative to noise.
  • Higher-Order Statistics SQI (SQI<sub>hos</sub>): Combines skewness and kurtosis.
  • Skewness (SQI<sub>skew</sub>): Measures the asymmetry of the signal's distribution.

They analyzed the performance of these SSQIs using various window sizes (1s, 2s, 5s, and 10s) across multiple publicly available ECG datasets. The evaluation focused on accuracy, sensitivity, and specificity. A key aspect of their methodology was the development of a semi-automatic tool to annotate ECG datasets, ensuring a more reliable ground truth for evaluating the SSQIs.

Datasets

The study utilized six publicly available ECG datasets from PhysioNet:

  • ECG-ID dataset
  • Tele ECG dataset
  • BIDMC Dataset
  • MIT/BIH arrhythmia dataset
  • Physionet/CINC 2011 dataset
  • Physionet/CINC 2014 dataset

The paper notes that this is a larger and more diverse set of datasets than typically used in previous SSQI validation studies.

Key Insights and Ideas

  • Dataset Diversity is King (and a Killer): The performance of SSQIs is significantly more sensitive to the specific dataset being analyzed than to the window size used for analysis. This means an index that works well on one dataset might perform poorly on another.
  • Low Specificity is a Major Hurdle: While many SSQIs can achieve high accuracy in identifying noise-free segments, their specificity (correctly identifying noisy segments) is consistently low. This bias towards classifying segments as noise-free is a critical limitation for clinical applications where misclassifying noise as a clean signal can be detrimental.
  • Static Thresholds are Problematic: The reliance on fixed thresholds for classifying signals as noisy or noise-free is identified as a primary reason for the performance fluctuations across datasets. A threshold that works for one dataset may not be appropriate for another.
  • No Single "Best" SSQI: The study suggests that no single SSQI is universally optimal across all datasets and conditions. Some indices (like SQI<sub>kur</sub> and SQI<sub>hos</sub>) show more consistent performance, while others (like SQI<sub>snr</sub>) can achieve very high accuracy on specific datasets but are highly variable.

Key Results

  • Accuracy vs. Specificity Trade-off: SSQIs generally showed high accuracy rates (e.g., SQI<sub>snr</sub> reaching 97.51% on the MIT/BIH arrhythmia dataset). However, their specificity was often very low, with some values dropping to near zero.
  • Window Size Impact is Minimal: The choice of window size (1s, 2s, 5s, 10s) had a relatively minor impact on the overall accuracy of most SSQIs compared to the dataset diversity.
  • SQI<sub>p</sub> Struggles: The relative power of QRS complex index (SQI<sub>p</sub>) generally showed the lowest accuracy across all datasets and window sizes, particularly with shorter windows. This is attributed to its frequency-specific nature.
  • SQI<sub>kur</sub> and SQI<sub>hos</sub> Show Consistency: Kurtosis (SQI<sub>kur</sub>) and higher-order statistics (SQI<sub>hos</sub>) demonstrated more consistent performance across different datasets compared to SQI<sub>p</sub> and SQI<sub>snr</sub>.

Limitations and Future Work

The authors acknowledge that the current fixed-threshold-based SSQIs are not robust enough for reliable clinical use due to their low specificity and dataset dependency. They propose several future directions:

  • Generalized Thresholds: Developing thresholds that are not dataset-specific but rather adaptive to signal features.
  • Machine Learning Approaches: Utilizing machine learning and deep learning models that can learn noise patterns from features or raw data, moving away from amplitude-based thresholds.
  • Adaptive Thresholds: Implementing dynamic thresholding mechanisms that adjust based on the characteristics of the ECG signal being analyzed.
  • Formal Definition of Noisy ECG: Collaborating with healthcare professionals to establish a more precise definition of what constitutes a "noisy" ECG signal.
  • Relationship between Signal Features and SSQIs: Further research into how specific ECG signal characteristics (like frequency content) influence SSQI performance.
  • Computational Cost: Considering computational efficiency and deployability for wearable devices when developing new algorithms.

Summary updated

Abstract

Electrocardiogram (ECG) signal quality indices (SQIs) are essential for improving diagnostic accuracy and reliability of ECG analysis systems. In various practical applications, the ECG signals are corrupted by different types of noise. These corrupted ECG signals often provide insufficient and incorrect information regarding a patient's health. To solve this problem, signal quality measurements should be made before an ECG signal is used for decision-making. This paper investigates the robustness of existing popular statistical signal quality indices (SSQIs): relative power of QRS complex (SQI<sub><i>p</i></sub>), skewness (SQI<sub>skew</sub>), signal-to-noise ratio (SQI<sub>snr</sub>), higher order statistics SQI (SQI<sub>hos</sub>) and peakedness of kurtosis (SQI<sub>kur</sub>). We analysed the robustness of these SSQIs against different window sizes across diverse datasets. Results showed that the performance of SSQIs considerably fluctuates against varying datasets, whereas the impact of varying window sizes was minimal. This fluctuation occurred due to the use of a static threshold value for classifying noise-free ECG signals from the raw ECG signals. Another drawback of these SSQIs is the bias towards noise-free ECG signals, that limits their usefulness in clinical settings. In summary, the fixed threshold-based SSQIs cannot be used as a robust noise detection system. In order to solve this fixed threshold problem, other techniques can be developed using adaptive thresholds and machine-learning mechanisms.
BibTeX
@article{rahman2022robustness,
  author = {Saifur Rahman and Chandan Karmakar and Iynkaran Natgunanathan and John Yearwood and Marimuthu Palaniswami},
  title = {{Robustness of electrocardiogram signal quality indices}},
  journal = {Journal of The Royal Society Interface},
  year = {2022},
  doi = {10.1098/rsif.2022.0012},
}
Figure