Exploring Finetuned Audio-LLM on Heart Murmur Features

Adrian Nicolas Florea, Xilin Jiang, Nima Mesgarani, Xiaofan JiangSmart 2025 · DOI · arXiv · OpenReview

TL;DR

This paper explores the potential of finetuning large language models (LLMs) for audio, specifically Qwen2-Audio, to analyze heart sounds (phonocardiograms or PCGs) for diagnosing cardiovascular diseases. By adapting an audio LLM with a segmentation pre-processing step, the authors achieve state-of-the-art performance in classifying multiple nuanced heart murmur features, even for rare "long-tail" cases.

What Problem Does This Paper Address?

Cardiovascular diseases are a leading cause of death, and listening to heart sounds with a stethoscope (auscultation) is a crucial diagnostic tool. Physicians analyze various features of heart murmurs – abnormal sounds indicating potential issues – such as their timing, grading, pitch, and quality. While existing deep learning models can classify basic murmur presence (healthy vs. unhealthy), they struggle with these more detailed, clinically significant features. This limits their utility as diagnostic aids. Furthermore, many existing models are not robust to variations in audio quality or recording methods, and they often fail to classify rare murmur types due to limited training data. This paper aims to bridge this gap by leveraging the advanced reasoning capabilities of audio LLMs to comprehensively analyze PCGs and predict a wider range of murmur features, improving diagnostic accuracy and robustness.

Methods

The proposed system combines two key components:

  1. PCG Segmentation Network (SSAMBA): To improve robustness and handle variations in audio, the authors employ a Mamba-based audio encoding model called SSAMBA. This model is used to segment the PCG recordings into distinct heartbeats and silences. This segmentation helps the LLM focus on relevant parts of the audio and can reduce the need for extensive labeled data.
  2. Finetuned Audio LLM (Qwen2-Audio): The core of the system is the Qwen2-Audio model, a powerful audio LLM that combines a Whisper-based audio encoder with a large language model (Qwen-7B). This model is finetuned on the PCG dataset to perform multi-feature classification. The finetuning process uses a full low-rank adaptation (LoRA) technique, which efficiently adapts the model by adding low-rank matrices to the original frozen weights of the audio encoder and LLM. The model is trained using a multiple-choice, single-answer approach to predict one of several predefined labels for each murmur feature.

Datasets

The primary dataset used is the PhysioNet CirCor DigiScope phonocardiogram (PCG) dataset. This dataset is described as the largest collection of pediatric heart sound recordings, containing over 312 hours of data from 1,568 patients. The recordings are meticulously annotated by experts for various murmur characteristics, including timing, shape, pitch, grading, quality, and location, as well as heart beat segmentation. The dataset is stratified by class (Present/Absent/Unknown) and resampled to 16 kHz, with a 75/25% split for training and testing.

For evaluation, the authors also use:

  • PhysioNet/CinC 2016 PCG dataset (training sets a through f) for zero-shot normal/abnormal classification.
  • Pascal dataset (sets A and B) for zero-shot normal/abnormal classification.

Key Insights and Ideas

  • LLMs for Biomedical Audio: This work demonstrates that LLMs, traditionally strong in text and speech, can be effectively adapted for complex biomedical audio analysis like PCG interpretation.
  • Segmentation as a Precursor: Using a dedicated segmentation model (SSAMBA) significantly enhances the robustness and generalizability of the LLM, especially for unseen datasets or varied recording conditions.
  • Beyond Binary Classification: The ability to predict multiple, nuanced murmur features (timing, grading, pitch, quality) is a significant step beyond simple healthy/unhealthy classification, offering richer diagnostic information.
  • Tackling Long-Tail Distributions: The finetuned LLM shows remarkable success in classifying "long-tail" murmur features, which have limited training data, a feat that previous methods have struggled with. This is crucial for diagnosing rarer conditions.
  • Roman Numeral Challenge: The model's difficulty with Roman numeral grading (e.g., i/vi) highlights a specific limitation related to the text encoder's frozen state and the need for more sophisticated handling of such symbolic representations.

Key Results

  • The proposed finetuned audio LLM system outperforms state-of-the-art methods in 8 out of 11 murmur features.
  • It performs comparably to existing methods in the remaining 3 features.
  • The system achieves 100% accuracy for systolic timing, shape, pitch, and quality, both with and without the segmentation frontend.
  • For diastolic features, the model achieves 100% accuracy for timing, pitch, and quality.
  • The segmentation frontend (W.S) improves diastolic grading performance above chance level.
  • In zero-shot normal/abnormal classification on benchmark datasets, the system shows significant improvements with the segmentation frontend, reaching 66.33% accuracy on the PhysioNet CirCor 2016 dataset.

Limitations and Future Work

  • Roman Numeral Understanding: The model struggles with Roman numeral grading (e.g., i/vi, ii/vi) due to the frozen text encoder and the symbolic nature of these labels. Future work could involve unfreezing parts of the text encoder or using alternative representations for these labels.
  • Diastolic Grading Performance: While improved by segmentation, diastolic grading still exhibits poor performance. Further investigation into this specific feature is warranted.
  • Data Augmentation/Training: The authors suggest that extra training with more murmur-present examples might further improve performance, especially for long-tail diastolic features.
  • Broader Feature Set: The paper calls for further research to expand the set of classified features and inspire advancements in comprehensive cardiac sound analysis.
  • Human Oversight: The paper emphasizes that while LLMs can assist, human cardiologists remain essential for clinical decision-making, requiring expert judgment and patient interaction.

Summary updated

Abstract

Large language models (LLMs) for audio have excelled in recognizing and analyzing human speech, music, and environmental sounds. However, their potential for understanding other types of sounds, particularly biomedical sounds, remains largely underexplored despite significant scientific interest. In this study, we focus on diagnosing cardiovascular diseases using phonocardiograms, i.e., heart sounds. Most existing deep neural network (DNN) paradigms are restricted to heart murmur classification (healthy vs unhealthy) and do not predict other acoustic features of the murmur such as timing, grading, harshness, pitch, and quality, which are important in helping physicians diagnose the underlying heart conditions. We propose to finetune an audio LLM, Qwen2-Audio, on the PhysioNet CirCor DigiScope phonocardiogram (PCG) dataset and evaluate its performance in classifying 11 expert-labeled murmur features. Additionally, we aim to achieve more noise-robust and generalizable system by exploring a preprocessing segmentation algorithm using an audio representation model, SSAMBA. Our results indicate that the LLM-based model outperforms state-of-the-art methods in 8 of the 11 features and performs comparably in the remaining 3. Moreover, the LLM successfully classifies long-tail murmur features with limited training data, a task that all previous methods have failed to classify. These findings underscore the potential of audio LLMs as assistants to human cardiologists in enhancing heart disease diagnosis.
BibTeX
@inproceedings{florea2025exploring,
  author = {Adrian Nicolas Florea and Xilin Jiang and Nima Mesgarani and Xiaofan Jiang},
  title = {{Exploring Finetuned Audio-LLM on Heart Murmur Features}},
  booktitle = {Smart},
  year = {2025},
  doi = {10.1016/j.smhl.2025.100557},
}
Figure