TL;DR
This paper introduces an event-based framework for evaluating automatic cough detection algorithms, moving beyond traditional sample-based metrics that are easily skewed by data imbalance. By focusing on the precise timing and duration of cough events, their proposed method offers a more clinically relevant assessment of algorithm performance, crucial for developing effective wearable cough monitoring systems.
What Problem Does This Paper Address?
Chronic cough is a prevalent condition that significantly impacts patients' quality of life. Currently, its assessment relies heavily on subjective patient questionnaires, which have limited correlation with actual cough counts. While wearable devices with machine learning (ML) algorithms offer a promising avenue for objective, continuous cough monitoring, a critical gap exists: existing evaluation metrics for these algorithms often don't align with what clinicians need. Many studies focus on classifying short audio segments as "cough" or "non-cough," but this doesn't directly translate to clinically meaningful endpoints like the number of cough events or their temporal patterns (e.g., cough bouts). Furthermore, common metrics like accuracy and specificity can be misleading when dealing with imbalanced datasets, where coughs are infrequent compared to non-cough sounds. This paper aims to bridge this gap by proposing and demonstrating an event-based evaluation framework that provides a more robust and clinically relevant assessment of cough counting algorithms.
Methods
The authors propose an event-based (EB) evaluation framework that focuses on identifying the precise start and end times of individual cough events. This contrasts with traditional sample-based (SB) metrics, which evaluate performance on fixed-length windows of data. To illustrate the shortcomings of SB metrics and the benefits of their EB approach, they implement a simple ML classifier (eXtreme Gradient Boosting - XGB) trained on audio data.
Key methodological aspects include:
- Data Segmentation and Feature Extraction: Audio features are extracted from data segments.
- Semi-Supervised Data Augmentation: To improve generalizability and handle class imbalance, the training dataset is augmented using cough samples from the COUGHVID dataset.
- Novel Post-Processing Algorithm: A crucial innovation is a post-processing algorithm that transforms the output of a standard sample-based classifier into an event-based detection. This algorithm uses hysteresis thresholding on signal power to identify cough spike regions, merges overlapping regions, and adapts start/end locations based on cough physiology (compressive, cough spike, and expiratory phases) and typical cough durations. This effectively converts a window-based classifier into one that can delineate individual cough events.
- Event-Based Scoring: The EB framework uses a scoring library adapted for cough detection, defining parameters like maximum event duration and tolerance thresholds based on cough physiology (coughs typically lasting 0.3-0.5s).
Datasets
The primary dataset used for training and evaluation is the "only known open dataset to delineate the start and stop times of each individual cough sound" [10]. This dataset includes audio and kinematic data from healthy subjects performing various vocalizations (forced coughs, breathing, throat clearing, speaking, laughing) under noisy conditions.
For evaluation, the test dataset is segmented into four different distributions to simulate varying class imbalance scenarios:
- Balanced-Silence (Bal-S): Balanced coughs and background noise/breathing.
- Balanced-Noise (Bal-N): Balanced coughs and sounds that can be confused for coughing.
- Example Experiment (Exp): Approximates data distributions reported in a prior state-of-the-art cough counting study.
- Whole Dataset (Whole): Uses all available segmented data with roughly equal distributions of all sound types.
The COUGHVID dataset is mentioned as being used for semi-supervised data augmentation during training.
Key Insights and Ideas
- Sample-based metrics are misleading: Traditional metrics like Accuracy (AC) and Negative Predictive Value (NPV) can be artificially inflated by class imbalance, showing better performance on datasets with fewer coughs. This is counterintuitive, as imbalanced datasets are precisely where robust evaluation is most needed.
- Specificity (SP) is often saturated: Due to the vast majority of non-cough samples, SP metrics tend to be high regardless of algorithm performance, making them less informative for practical cough detection.
- Event-based metrics are clinically relevant: Focusing on the precise timing and duration of cough events aligns better with clinical endpoints like cough bouts and temporal patterns, providing more actionable insights for diagnosis and treatment monitoring.
- Window length significantly impacts SB performance: Longer windows in SB models can lead to undercounting of coughs, as multiple coughs within a single window might be treated as one. EB models, with their post-processing, are less susceptible to this distortion.
- Post-processing is key to EB: The paper demonstrates that a standard sample-based classifier can be effectively transformed into an event-based detector through a physiology-inspired post-processing step.
Key Results
- Metric Sensitivity to Imbalance: Performance metrics like Precision (PR) and False Positives per hour (FP/hr) varied significantly across different dataset distributions, highlighting the impact of class imbalance. For instance, PR decreased and FP/hr increased as non-cough samples became more prevalent.
- EB vs. SB Performance: The EB model showed higher Sensitivity (SE) (73.1%) compared to the SB classifier (57.4%), particularly in capturing cough bursts. The EB model also better segmented coughs within longer windows, avoiding the undercounting seen in SB models.
- Window Length Impact: While the EB model's TP count was relatively stable across different window lengths, the SB classifier showed a significant decrease in TPs as window length increased.
- Recommended Metrics: The authors strongly recommend using Sensitivity (SE), Precision (PR), F-1 Score (F1), and False Positives per hour (FP/hr) for evaluating cough counting algorithms, as these are less susceptible to class imbalance and more clinically relevant.
Limitations and Future Work
- Dataset Limitations: A significant limitation is that the algorithms are trained and tested on forced coughs from healthy subjects. The authors emphasize the need for an open, labeled, clinical dataset from patients with chronic cough disorders to better compare algorithm performance in real-world scenarios.
- Generalizability: While the proposed framework is designed to be generalizable, its application to other respiratory disorder symptoms (like wheezing or stridor) would require further validation.
- Future Work: The authors suggest extending their event-based framework to other respiratory symptoms and encourage the research community to utilize their open-source repository for further development and testing. They also highlight the need for clear reporting of dataset characteristics (cough-to-non-cough ratio, sample contents) when performance claims are made.
Summary updated