Feature-Based Fusion Using CNN for Lung and Heart Sound Classification

Zeenat Tariq, Sayed Khushal Shah, Yugyung Lee

TL;DR

This paper introduces FDC-FS, a novel deep learning framework that fuses multiple Convolutional Neural Network (CNN) models to classify lung and heart sounds. By transforming audio features into images and combining specialized CNNs, FDC-FS achieves state-of-the-art accuracy in detecting respiratory and cardiovascular conditions, even with noisy and imbalanced datasets.

What Problem Does This Paper Address?

Diagnosing lung and heart conditions through auscultation (listening with a stethoscope) is a crucial but challenging task. The complexity of audio data, its dynamic nature, and issues like noise and data imbalance make automated classification difficult. Existing deep learning approaches often struggle with limited or unbalanced datasets, hindering their real-world applicability. This paper aims to overcome these limitations by developing a robust and accurate system for classifying lung and heart sounds.

Methods

The core of the FDC-FS framework involves several key steps:

  • Feature Extraction and Transformation: Three distinct audio features – Spectrogram, Mel-Frequency Cepstral Coefficients (MFCC), and Chromagram – are extracted from the audio signals. These features are then transformed into image representations.
  • Data Augmentation: To address data scarcity and imbalance, techniques like noise distortion, pitch shifting, and time stretching are applied to the audio data.
  • Multi-Model Architecture: Three specialized CNN models (FDC-1, FDC-2, and FDC-3) are designed, each optimized for learning from one of the extracted image features.
  • Feature-Based Fusion: The outputs of these three specialized CNN models are fused in a final model (FDC-FS). This fusion leverages transfer learning from the individual models to create a more powerful and generalized classifier.

Datasets

The study utilizes two publicly available datasets:

  • Lung Sound Dataset: The ICHBI 2017 challenge dataset, containing 920 annotated lung sound recordings.
  • Heart Sound Dataset: A heart challenge dataset, comprising 656 audio recordings for various heart sound categories.

The paper also mentions data augmentation techniques applied to these datasets, significantly increasing their size for training.

Key Insights and Ideas

  • Audio-to-Image Transformation is Key: Converting complex audio features into visual representations (images) allows CNNs, which excel at image analysis, to effectively learn patterns for sound classification.
  • Fusion of Specialized Models Outperforms Single Models: Combining the strengths of multiple CNNs, each trained on a specific feature type, leads to superior performance compared to individual models. This suggests that different features capture complementary information.
  • Data Augmentation is Crucial for Robustness: Techniques like noise injection and time stretching are vital for improving model performance, especially when dealing with real-world noisy and imbalanced medical audio data.
  • Transfer Learning Enhances Performance: Leveraging pre-trained models (implicitly through the fusion of FDC-1, FDC-2, and FDC-3) allows the FDC-FS model to learn more effectively and achieve higher accuracy.

Key Results

The FDC-FS model demonstrated state-of-the-art performance on both lung and heart sound classification tasks:

  • Lung Sound Classification:
    • Original Data: Achieved up to 97% accuracy with Spectrogram.
  • Augmented Data: Reached an impressive 99.1% accuracy with Spectrogram, 99% with MFCC, and 98.4% with Chromagram.
  • Heart Sound Classification:
    • Original Data: Achieved up to 93% accuracy with MFCC.
  • Augmented Data: Reached 97% accuracy with Spectrogram and Chromagram, and 96% with MFCC.

The paper highlights that FDC-FS achieved these high accuracies with a relatively low number of trainable parameters and a reduced number of epochs compared to some other state-of-the-art methods.

Limitations and Future Work

  • Generalization: The proposed framework performs well on lung and heart sound domains but may lack generalization to other audio classification tasks.
  • Interpretability: The authors acknowledge the need for interpretable deep learning models and explainable AI to provide evidence of unique patterns discovered for specific conditions.
  • Future Directions:
    • Developing suitable pattern mining methods for automatic network design.
  • Improving transfer learning and knowledge distillation for better fusion.
  • Enhancing pre-processing and data augmentation techniques to handle noise and imbalance more effectively.
  • Investigating the impact of floating window size on model efficiency.
  • Extending the work to multi-task classification combining lung and heart models.
  • Applying the models to more diverse datasets.

Summary updated

Abstract

Lung or heart sound classification is challenging due to the complex nature of audio data, its dynamic properties of time, and frequency domains. It is also very difficult to detect lung or heart conditions with small amounts of data or unbalanced and high noise in data. Furthermore, the quality of data is a considerable pitfall for improving the performance of deep learning. In this paper, we propose a novel feature-based fusion network called FDC-FS for classifying heart and lung sounds. The FDC-FS framework aims to effectively transfer learning from three different deep neural network models built from audio datasets. The innovation of the proposed transfer learning relies on the transformation from audio data to image vectors and from three specific models to one fused model that would be more suitable for deep learning. We used two publicly available datasets for this study, i.e., lung sound data from ICHBI 2017 challenge and heart challenge data. We applied data augmentation techniques, such as noise distortion, pitch shift, and time stretching, dealing with some data issues in these datasets. Importantly, we extracted three unique features from the audio samples, i.e., Spectrogram, MFCC, and Chromagram. Finally, we built a fusion of three optimal convolutional neural network models by feeding the image feature vectors transformed from audio features. We confirmed the superiority of the proposed fusion model compared to the state-of-the-art works. The highest accuracy we achieved with FDC-FS is 99.1% with Spectrogram-based lung sound classification while 97% for Spectrogram and Chromagram based heart sound classification.
BibTeX
@inproceedings{tariq2022feature,
  author = {Zeenat Tariq and Sayed Khushal Shah and Yugyung Lee},
  title = {{Feature-Based Fusion Using CNN for Lung and Heart Sound Classification}},
  booktitle = {Sensors},
  year = {2022},
  url = {https://openreview.net/forum?id=Dv2cLWCxfwp},
}
Figure