Abstract
Detecting medical conditions from speech acoustics is fundamentally a weakly-supervised learning problem: a single, often noisy, session-level label must be linked to nuanced patterns within a long, complex audio recording. This task is further hampered by severe data scarcity and the subjective nature of clinical annotations. While semi-supervised learning (SSL) offers a viable path to leverage unlabeled data, existing audio methods often fail to address the core challenge that pathological traits are not uniformly expressed in a patient's speech. We propose a novel, audio-only SSL framework that explicitly models this hierarchy by jointly learning from frame-level, segment-level, and session-level representations within unsegmented clinical dialogues. Our end-to-end approach dynamically aggregates these multi-granularity features and generates high-quality pseudo-labels to efficiently utilize unlabeled data. Extensive experiments show the framework is model-agnostic, robust across languages and conditions, and highly data-efficient—achieving, for instance, 90% of fully-supervised performance using only 11 labeled samples.
Introduction
Detecting disease from speech is often described as a "needle in a haystack" problem. In a typical clinical interview, a practitioner provides a single label (e.g., "depressed") for an entire 20-minute conversation. This creates a scenario of weak or far-end supervision. The primary challenges include data scarcity due to the high cost of expert labels, sparse manifestation where symptoms appear sporadically, and a granularity mismatch between short-term audio features and long-term clinical diagnoses. Existing methods frequently treat every segment of a recording as if it carries the disease signature equally, which is rarely accurate. This research aims to bridge the gap between fine-grained audio features and high-level clinical diagnoses without requiring massive amounts of labeled data.
Methods: Multi-Level Data Modeling
The proposed framework operates across three granularities simultaneously to capture the hierarchical nature of speech:
- Session-Level: A Transformer aggregates features from the entire dialogue to produce the final diagnosis.
- Clip-Level: A Recurrent Neural Network (RNN) models individual utterances. The model generates "pseudo-labels" for these clips during training, identifying specific parts of a conversation most indicative of a disease.
- Frame-Level: A Siamese network and contrastive loss ensure the model captures consistent acoustic patterns at the millisecond level, independent of high-level labels.
The system utilizes a Teacher-Student framework. A "Teacher" model, updated via Exponential Moving Average (EMA), generates high-quality pseudo-labels for unlabeled data, which the "Student" model then learns from. This creates a self-improving loop that thrives on unlabeled recordings.
Experimental Setup
The framework was evaluated on two distinct datasets:
- EATD-Corpus: A Chinese-language dataset for Depression Detection containing interviews with 162 participants.
- ADReSSo21: An English-language dataset for Alzheimer’s Detection featuring 237 samples of participants describing a picture.
The model is designed to be model-agnostic, working with various audio encoders such as wav2vec2, HuBERT, and WavLM.
Results and Discussion
The multi-level approach demonstrates significant performance gains and extreme data efficiency. On the depression task, the model achieved 90% of the performance of a fully supervised system using only 10% of the labels (approximately 11 samples).
Key insights from the experiments include:
- Hierarchy is Key: The frame-level component provided the most significant gains by forcing the model to understand low-level acoustic consistency, making it robust against noisy session-level labels.
- Pseudo-Label Evolution: As training progresses, the quality of the model’s pseudo-labels on unlabeled data can surpass the quality of noisy human labels in low-data scenarios.
- No Diarization Needed: The model performs well on raw, multi-speaker dialogues without requiring the removal of the interviewer's voice, automatically focusing on the patient's salient segments.
Conclusion
This work provides a principled approach to learning from weak, far-end supervision in medical speech analysis. While currently audio-only, future work could incorporate multimodal approaches (combining audio and text) and specialized pre-training for clinical contexts to further bridge the performance gap. The framework's ability to achieve high accuracy with minimal labeled data makes it a promising tool for scalable medical screening.
Summary updated