Abstract
Computational paralinguistics often relies on subjective human annotations, which can introduce significant noise into training data. This paper presents a discriminative classification model that explicitly accounts for label reliability by treating it as a latent variable. Using a mixture-of-experts framework, the model distinguishes between reliable samples and noisy outliers. Evaluation on a pathological speech corpus demonstrates improved classification accuracy, particularly when acoustic features are inherently noisy or less informative.
Introduction
In the study of speech pathologies, "intelligibility" is a critical metric often determined by human listeners. However, human judgment is subjective and prone to inconsistency, especially when dealing with severely degraded speech from patients with head and neck cancers. Standard machine learning approaches typically treat these human-provided labels as absolute ground truth, which can lead to overfitting on noise. This research addresses the challenge of training robust classifiers when both input features (acoustic data) and target labels (human judgment) are prone to error.
Methodology: The Discriminative Reliability-Aware Model
The proposed approach introduces a Discriminative Reliability-Aware Model that incorporates a hidden binary variable, $R$, representing the reliability of a given sample.
Model Architecture
The system is structured as a mixture-of-experts (MoE) model consisting of two primary components:
- The Reliable Expert: A Maximum Entropy (MaxEnt) model that utilizes acoustic features to predict the target class when $R=1$.
- The Unreliable Expert: A feature-independent "garbage model" that captures label noise when $R=0$, essentially predicting based on the prior class distribution.
The model learns to predict the probability of reliability ($R$) directly from the input features using a logistic regression component. Parameters are optimized using the Expectation-Maximization (EM) algorithm, where the E-step estimates sample reliability and the M-step updates the classifier and reliability predictor to better fit the data.
Experimental Setup
The model was validated using the NKI CCRT Speech Corpus, containing 2,385 sentence-level utterances from 55 speakers undergoing treatment for head and neck tumors. Utterances were labeled for binary intelligibility by human annotators. The feature set included 13 acoustic parameters categorized into three domains: Prosody (rhythm/pitch), Voice Quality (breathiness/harshness), and Pronunciation.
Results and Discussion
The reliability-aware model was compared against a standard "reliability-blind" logistic regression baseline. The results indicate that the proposed model is most effective when the underlying features are less informative or "messier."
As shown in the results, the model achieved performance gains for the Voice Quality and Pronunciation feature sets, which exhibited lower average reliability ($\rho_{avg}$). In these cases, the model successfully identified and down-weighted unreliable samples, allowing the "reliable expert" to focus on clear patterns. Conversely, for the Prosody set—which was inherently more reliable—the baseline performed slightly better, indicating that the added complexity of modeling latent reliability is most beneficial in high-noise scenarios.
Conclusion
This work demonstrates that explicitly modeling the reliability of human annotations can enhance the robustness of speech classification systems. By learning which regions of the feature space are prone to subjective error, the model can effectively filter out noise. Future research will explore multi-view reliability modeling and the integration of raw labels from multiple annotators to further refine the estimation of subjective bias.
Summary updated