Abstract
The lack of labeled data is a common challenge in speech classification tasks, particularly those requiring extensive subjective assessment, such as cognitive state classification. In this work, we propose a Semi-Supervised Learning (SSL) framework, introducing a novel multi-view pseudo-labeling method that leverages both acoustic and linguistic characteristics to select the most confident data for training the classification model. Acoustically, unlabeled data are compared to labeled data using the Frechet audio distance, calculated from embeddings generated by multiple audio encoders. Linguistically, large language models are prompted to revise automatic speech recognition transcriptions and predict labels based on our proposed task-specific knowledge. High-confidence data are identified when pseudo-labels from both sources align, while mismatches are treated as low-confidence data. A bimodal classifier is then trained to iteratively label the low-confidence data until a predefined criterion is met. We evaluate our SSL framework on emotion recognition and dementia detection tasks. Experimental results demonstrate that our method achieves competitive performance compared to fully supervised learning using only 30% of the labeled data and significantly outperforms two selected baselines.
Introduction
Classifying cognitive states—such as a speaker's emotional state or signs of dementia—is notoriously difficult because labeling the data requires expert subjective assessment. Unlike transcribing words, which is relatively straightforward, determining if a speaker sounds "frustrated" or shows "early-stage cognitive decline" is time-consuming and expensive. This creates a massive data bottleneck: while unlabeled speech data is abundant, high-quality labeled data for these specific tasks is scarce. Traditional Semi-Supervised Learning (SSL) attempts to bridge this gap through pseudo-labeling, but these "guesses" are often noisy. This paper introduces a framework designed to make pseudo-labeling significantly more reliable by leveraging multi-modal insights.
Methodology
The proposed framework introduces a Multi-View Pseudo-Labeling approach that evaluates speech through two distinct paths:
1. The Acoustic Path
Instead of simple point-to-point comparisons, the system utilizes Fréchet Audio Distance (FAD). It calculates the statistical distance between the distributions of labeled and unlabeled data across four different audio encoders: VGGish, EnCodec, Wav2vec 2.0, and CLAP. If an unlabeled sample is acoustically similar to a specific labeled group (e.g., "Angry"), it receives that pseudo-label.
2. The Linguistic Path
The system employs a pipeline called REVISE-REASON-RECOGNIZE (R3):
- Revise: An LLM corrects errors in raw Automatic Speech Recognition (ASR) transcripts.
- Reason: The LLM explains why a label might apply based on task-specific knowledge, such as linguistic "triggers" or acoustic descriptors like pitch and jitter.
- Recognize: The LLM predicts the final label.
Data Selection and Training
Data is only considered "high-confidence" if the acoustic and linguistic paths agree on the label. These samples are then used to train a bimodal classifier (combining HuBERT for audio and RoBERTa for text) in an iterative process. The researchers found that "Modality-gated fusion"—where the model dynamically weights audio versus text—performed best, mitigating issues like ASR errors or background noise.
Experimental Results
The framework was evaluated on two gold-standard benchmarks: IEMOCAP (Emotion Recognition) and ADReSSo (Dementia Detection).
Efficiency and Baseline Comparison
The method demonstrated remarkable efficiency, achieving performance competitive with a fully supervised model (using 100% of labels) while using only 30% of the labeled data. As shown in the results, the proposed method significantly outperformed standard SSL techniques such as "Co-training" and "Decision Merging."
In dementia detection with 30% labels, the method achieved approximately 80.87% accuracy, compared to the best baseline of 74.73%. Furthermore, the performance remained stable even when labeled data was reduced to 20%, a scenario where standard supervised models typically fail.
Conclusion
By combining acoustic similarity metrics with LLM-based reasoning, this multi-view framework effectively filters noise in pseudo-labeling. While the system remains somewhat dependent on ASR quality and involves higher computational costs due to the use of multiple encoders and LLMs, it provides a robust path forward for cognitive state classification in low-resource environments. Future work may explore more granular stages of cognitive decline and more complex emotional nuances.
Summary updated