Medical Speech Symptoms Classification via Disentangled Representation

Jianzong Wang, Pengcheng Li, Xulong Zhang, Ning Cheng, Jing Xiao

Abstract

Accurate classification of medical symptoms from patient speech requires capturing both linguistic content and paralinguistic intent. Traditional models often struggle to separate these features, leading to suboptimal performance when speech-to-text transcriptions are noisy. This paper presents DRSC (Disentangled Representation for Speech Classification), a multi-modal framework that utilizes a Generative Adversarial Network (GAN) to disentangle symptom-related intent from general speech style. By enforcing domain-invariance through a "Cross and Restore" mechanism, DRSC achieves a state-of-the-art accuracy of 95.58% on clinical datasets and demonstrates significant robustness to transcription errors.

Introduction

In clinical diagnostics, a patient’s vocalization provides a rich, multi-layered data stream. Beyond the literal meaning of the words spoken (content), the acoustic properties—such as pitch, rhythm, and vocal strain—convey critical information about their physiological and psychological state (intent). While modern healthcare increasingly looks toward AI for automated symptom screening, current systems often rely on a "pipeline" approach: converting speech to text via Automatic Speech Recognition (ASR) and then analyzing the text. This method inherently discards non-verbal cues.

Multi-modal models that attempt to combine audio and text often suffer from feature interference, where the unique "noise" of an individual's speaking style obscures the "signal" of the medical symptom. To address this, the researchers propose the Disentangled Representation for Speech Classification (DRSC) model, which explicitly separates intent from content to improve diagnostic precision.

Methodology: The DRSC Framework

The DRSC model is built upon a GAN-based architecture designed to extract domain-invariant intent representations from both text and audio domains.

Dual Encoding and Disentanglement

The architecture employs two sets of encoders for the text and Mel-spectrogram (audio) domains. Each domain contains:

  • Content Encoder ($E_c$): Captures domain-specific style and linguistic structure.
  • Intent Encoder ($E_i$): Extracts features specifically related to the underlying medical symptoms.

The "Cross and Restore" Operation

To ensure that the intent encoders are capturing truly domain-invariant information, the model utilizes a "Cross and Restore" mechanism. In this process, the intent representation from the text domain is paired with the content representation from the audio domain (and vice versa). The model then attempts to reconstruct the original signals. Successful reconstruction proves that the intent representation is independent of the source domain, effectively stripping away the "content" noise.

Feature Fusion and Loss Functions

Once disentangled, the intent representations from both modalities are fused into a single joint representation for classification. The training process is guided by a multi-faceted objective function comprising six losses, including cycle-consistency loss for reconstruction accuracy and adversarial loss to ensure the generated features are indistinguishable from real data.

Experimental Evaluation

The model was evaluated using the Medical Speech, Transcription, and Intent dataset, consisting of 6,661 audio segments across 25 symptom categories, such as "headache," "stomach ache," and "joint pain."

Robustness to ASR Noise

A critical test of the DRSC framework was its performance under "imprecise" conditions. Using transcriptions from a pre-trained ASR model with a 26% word error rate, the researchers observed only a marginal 4% drop in accuracy. This suggests that the disentangled acoustic intent encoders successfully compensate for linguistic gaps in the text.

Results and Discussion

DRSC achieved a peak average accuracy of 95.58%, a substantial improvement over the multi-modal baseline (SpeechIC), which reached only 82.47%. Ablation studies further highlighted the importance of the GAN-based training; removing the auxiliary loss functions (VAE, Latent Regression, and Adversarial losses) resulted in a performance drop to 81.19%.

The choice of distance metric also proved pivotal. The researchers found that L1 (Manhattan distance) outperformed L2 and cosine distance as the criterion for the loss functions, suggesting that the intent space is better modeled by linear differences in feature vectors.

Conclusion and Future Work

The DRSC model demonstrates that disentangling intent from content is a powerful strategy for medical speech classification. By creating a shared intent space between text and audio, the model achieves high accuracy and resilience to transcription errors.

Future research will focus on addressing confusion between symptoms with similar acoustic profiles (e.g., joint pain) and testing the model on "wild" clinical data characterized by higher background noise and diverse dialects. Additionally, optimizing the GAN architecture for real-time deployment on mobile devices remains a key objective for clinical accessibility.

Summary updated

BibTeX
@inproceedings{wang2024medical,
  author = {Jianzong Wang and Pengcheng Li and Xulong Zhang and Ning Cheng and Jing Xiao},
  title = {{Medical Speech Symptoms Classification via Disentangled Representation}},
  booktitle = {CSCWD},
  year = {2024},
  doi = {10.1109/cscwd61410.2024.10580785},
}
Figure