Semi-Supervised Diseased Detection from Speech Dialogues with Multi-Level Data Modeling

Xingyuan Li, Weizhi Ma

Abstract

Detecting medical conditions from speech acoustics is fundamentally a weakly-supervised learning problem: a single, often noisy, session-level label must be linked to nuanced patterns within a long, complex audio recording. This task is further hampered by severe data scarcity and the subjective nature of clinical annotations. While semi-supervised learning (SSL) offers a viable path to leverage unlabeled data, existing audio methods often fail to address the core challenge that pathological traits are not uniformly expressed in a patient's speech. We propose a novel, audio-only SSL framework that explicitly models this hierarchy by jointly learning from frame-level, segment-level, and session-level representations within unsegmented clinical dialogues. Our end-to-end approach dynamically aggregates these multi-granularity features and generates high-quality pseudo-labels to efficiently utilize unlabeled data. Extensive experiments show the framework is model-agnostic, robust across languages and conditions, and highly data-efficient—achieving, for instance, 90% of fully-supervised performance using only 11 labeled samples.

Introduction

Detecting disease from speech is often described as a "needle in a haystack" problem. In a typical clinical interview, a practitioner provides a single label (e.g., "depressed") for an entire 20-minute conversation. This creates a scenario of weak or far-end supervision. The primary challenges include data scarcity due to the high cost of expert labels, sparse manifestation where symptoms appear sporadically, and a granularity mismatch between short-term audio features and long-term clinical diagnoses. Existing methods frequently treat every segment of a recording as if it carries the disease signature equally, which is rarely accurate. This research aims to bridge the gap between fine-grained audio features and high-level clinical diagnoses without requiring massive amounts of labeled data.

Methods: Multi-Level Data Modeling

The proposed framework operates across three granularities simultaneously to capture the hierarchical nature of speech:

  • Session-Level: A Transformer aggregates features from the entire dialogue to produce the final diagnosis.
  • Clip-Level: A Recurrent Neural Network (RNN) models individual utterances. The model generates "pseudo-labels" for these clips during training, identifying specific parts of a conversation most indicative of a disease.
  • Frame-Level: A Siamese network and contrastive loss ensure the model captures consistent acoustic patterns at the millisecond level, independent of high-level labels.
The Multi-Level Data Modeling framework featuring Student and Teacher models across session, clip, and frame granularities.

The system utilizes a Teacher-Student framework. A "Teacher" model, updated via Exponential Moving Average (EMA), generates high-quality pseudo-labels for unlabeled data, which the "Student" model then learns from. This creates a self-improving loop that thrives on unlabeled recordings.

Experimental Setup

The framework was evaluated on two distinct datasets:

  1. EATD-Corpus: A Chinese-language dataset for Depression Detection containing interviews with 162 participants.
  2. ADReSSo21: An English-language dataset for Alzheimer’s Detection featuring 237 samples of participants describing a picture.

The model is designed to be model-agnostic, working with various audio encoders such as wav2vec2, HuBERT, and WavLM.

Results and Discussion

The multi-level approach demonstrates significant performance gains and extreme data efficiency. On the depression task, the model achieved 90% of the performance of a fully supervised system using only 10% of the labels (approximately 11 samples).

Performance comparison of the proposed method against baselines for Depression and Alzheimer's detection across various labeled data ratios.

Key insights from the experiments include:

  • Hierarchy is Key: The frame-level component provided the most significant gains by forcing the model to understand low-level acoustic consistency, making it robust against noisy session-level labels.
  • Pseudo-Label Evolution: As training progresses, the quality of the model’s pseudo-labels on unlabeled data can surpass the quality of noisy human labels in low-data scenarios.
  • No Diarization Needed: The model performs well on raw, multi-speaker dialogues without requiring the removal of the interviewer's voice, automatically focusing on the patient's salient segments.

Conclusion

This work provides a principled approach to learning from weak, far-end supervision in medical speech analysis. While currently audio-only, future work could incorporate multimodal approaches (combining audio and text) and specialized pre-training for clinical contexts to further bridge the performance gap. The framework's ability to achieve high accuracy with minimal labeled data makes it a promising tool for scalable medical screening.

Summary updated

Abstract

Detecting medical conditions from speech acoustics is fundamentally a weakly-supervised learning problem: a single, often noisy, session-level label must be linked to nuanced patterns within a long, complex audio recording. This task is further hampered by severe data scarcity and the subjective nature of clinical annotations. While semi-supervised learning (SSL) offers a viable path to leverage unlabeled data, existing audio methods often fail to address the core challenge that pathological traits are not uniformly expressed in a patient's speech. We propose a novel, audio-only SSL framework that explicitly models this hierarchy by jointly learning from frame-level, segment-level, and session-level representations within unsegmented clinical dialogues. Our end-to-end approach dynamically aggregates these multi-granularity features and generates high-quality pseudo-labels to efficiently utilize unlabeled data. Extensive experiments show the framework is model-agnostic, robust across languages and conditions, and highly data-efficient-achieving, for instance, 90\% of fully-supervised performance using only 11 labeled samples. This work provides a principled approach to learning from weak, far-end supervision in medical speech analysis.
BibTeX
@inproceedings{li2026semi,
  author = {Xingyuan Li and Weizhi Ma},
  title = {{Semi-Supervised Diseased Detection from Speech Dialogues with Multi-Level Data Modeling}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1194},
}
Figure