Rene: A Pre-trained Multi-modal Architecture for Auscultation of Respiratory Diseases

Pengfei Zhang, Zhihang Zheng, Shichen Zhang, Minghao Yang, Shaojun TangarXiv 2024 · DOI · arXiv

Overview

Rene is a pioneering multimodal deep-learning framework that adapts large-scale speech recognition models (specifically OpenAI’s Whisper) for clinical respiratory sound analysis. By fusing audio features with structured patient medical records, Rene achieves up to a 23% performance boost over existing benchmarks and is optimized for real-time deployment on Edge AI devices.

Introduction

Pulmonary auscultation—listening to lung sounds with a stethoscope—is a fundamental, non-invasive diagnostic tool. However, it is inherently subjective, requiring years of clinical experience to master. While previous AI models have attempted to automate this, they often face three major hurdles:

  1. Lack of Context: Most models focus solely on audio, ignoring the "clinical picture" (e.g., age, BMI, medical history) that doctors use to refine a diagnosis.
  2. Interpretability: Deep learning "black boxes" are difficult to trust in a clinical setting.
  3. Latency: Many high-performing models require cloud computing, which raises privacy concerns and prevents real-time feedback during a physical exam.

Methodology

The authors introduce a dual-stream architecture named after René Laennec, the inventor of the stethoscope.

Figure 1: The Rene dual-stream architecture, featuring the Clinical Record Module and the Feature Extraction Module.

Dual-Stream Architecture

  • The Audio Stream: Instead of building from scratch, the team fine-tuned Whisper, a large-scale pre-trained speech model. They enhanced this backbone with a Conformer architecture (combining CNNs for local patterns and Transformers for global context) and a Bi-GRU decoder to capture long-term temporal trends in breathing.
  • The Clinical Stream: Structured Electronic Medical Record (EMR) data is processed using LightGBM, a gradient-boosting framework. This allows the model to weigh factors like age and BMI alongside the audio.
  • Multimodal Fusion: The two streams are merged using a confidence-weighted parameter ($\alpha$). This allows the system to balance the "weight" of the audio versus the medical history.
  • Edge AI Optimization: To make the model "portable," the researchers developed Rene (S), a compressed 79M-parameter version designed to run locally on digital stethoscopes or mobile terminals using a dual-thread approach (one thread for recording, one for "on-the-fly" decoding).

Datasets

The study utilized a massive consolidated dataset (SFTData) comprising 16,300 training and 7,979 testing recordings, drawn from:

  • SPRSound: A pediatric respiratory sound database from Shanghai Jiao Tong University.
  • ICBHI: A large-scale benchmark featuring 5.5 hours of annotated respiratory cycles (crackles and wheezes).
  • KAUH: Data from King Abdulaziz University Hospital to ensure better generalization across diverse populations.

Key Insights

Transfer Learning from Speech to Lung Sounds

A major finding is that large-scale models trained on human speech (Whisper) possess a latent understanding of acoustics that translates remarkably well to respiratory sounds after fine-tuning. The Word Error Rate (WER) for lung sound classification dropped from ~127 to ~25 after fine-tuning.

Contextual Importance in Diagnosis

The researchers found that the model performs best when $\alpha = 0.2$, meaning it leans more heavily on medical records than audio for certain disease predictions. This mirrors clinical reality: a "crackle" in a 70-year-old with a high BMI means something very different than the same sound in a healthy toddler.

The Rene Trial Block

The authors implemented a "Rene Trial Block" using a triadic branch jump design. This progressively constricts the receptive field on one side while broadening it on the other, helping the model learn features at multiple scales simultaneously.

Figure 2: The Rene Trial Block architecture (left) and LightGBM feature importance analysis (right), demonstrating the multi-scale feature learning and the impact of clinical context.

Results

Rene significantly outperformed existing state-of-the-art models across multiple benchmarks:

  • SPRSound Database: Achieved improvements of 10.27% to 18.90% in event detection and audio classification tasks.
  • ICBHI Database: Disease prediction accuracy improved by 23% over the baseline in both mean average and harmonic scores.
  • Efficiency: The Rene (S) model maintains high accuracy while being small enough for real-time inference on edge devices, enabling rapid diagnostic responses without cloud dependency.

Limitations and Future Work

While Rene is a leap forward, the authors note several areas for expansion:

  • Missing Modalities: The current version does not yet integrate medical imaging (CT or X-ray) data, which could further bolster diagnostic reliability.
  • Data Scarcity: To combat the difficulty of collecting rare pathological sounds, the authors suggest using Generative Adversarial Networks (GANs) to simulate high-fidelity synthetic lung sounds for training.
  • LLM Synergy: Future iterations may integrate Large Language Models (like GPT-4) to provide natural language clinical decision support, explaining why a specific diagnosis was reached.

Summary updated

Abstract

Compared with invasive examinations that require tissue sampling, respiratory sound testing is a non-invasive examination method that is safer and easier for patients to accept. In this study, we introduce Rene, a pioneering large-scale model tailored for respiratory sound recognition. Rene has been rigorously fine-tuned with an extensive dataset featuring a broad array of respiratory audio samples, targeting disease detection, sound pattern classification, and event identification. Our innovative approach applies a pre-trained speech recognition model to process respiratory sounds, augmented with patient medical records. The resulting multi-modal deep-learning framework addresses interpretability and real-time diagnostic challenges that have hindered previous respiratory-focused models. Benchmark comparisons reveal that Rene significantly outperforms existing models, achieving improvements of 10.27%, 16.15%, 15.29%, and 18.90% in respiratory event detection and audio classification on the SPRSound database. Disease prediction accuracy on the ICBHI database improved by 23% over the baseline in both mean average and harmonic scores. Moreover, we have developed a real-time respiratory sound discrimination system utilizing the Rene architecture. Employing state-of-the-art Edge AI technology, this system enables rapid and accurate responses for respiratory sound auscultation(https://github.com/zpforlove/Rene).
BibTeX
@misc{zhang2024rene,
  author = {Pengfei Zhang and Zhihang Zheng and Shichen Zhang and Minghao Yang and Shaojun Tang},
  title = {{Rene: A Pre-trained Multi-modal Architecture for Auscultation of Respiratory Diseases}},
  year = {2024},
  eprint = {2405.07442},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2405.07442},
}
Figure