Discriminative Training Sensitivity to Label Noise

49 concepts42 papers4 proposals

In the pursuit of scaling audio and speech models, researchers have increasingly turned to massive, web-scraped, or automatically transcribed datasets. While these datasets provide the volume necessary for deep learning, they are frequently "noisy"—containing incorrect transcripts, mislabeled instrument sources, or coarse annotations that miss fine-grained details.

A recurring finding across speech and audio research is that discriminative training objectives are significantly more sensitive to this noise than standard generative or maximum likelihood approaches. While discriminative methods often yield superior performance on clean data, they can actively degrade a model when labels are imperfect.

The Margin of Error

The core intuition behind this sensitivity lies in how different objectives treat the relationship between a signal and its label. Standard Maximum Likelihood Estimation (MLE) aims to maximize the probability of the observed data. If a small portion of the labels is incorrect, the model essentially sees a slightly noisier distribution to fit.

In contrast, discriminative objectives—such as Minimum Phone Error (MPE) or Maximum Mutual Information (MMI)—are designed to maximize the margin between the "correct" hypothesis and all other competing hypotheses. As Kai Yu et al. observe, these objectives optimize the distinction between sequences. When a machine-generated transcript contains an error, a discriminative objective forces the model to treat that error as the absolute truth to be separated from the actual acoustic signal. This reinforces incorrect patterns, causing the model to learn decision boundaries that do not exist in reality.

Dong Yu et al. demonstrate that while generative models might see marginal performance drops from transcription errors, discriminative models can suffer substantial losses. This creates a "performance ceiling" where the very techniques intended to sharpen a model's accuracy instead amplify the flaws in the training data.

Sources of Noise in Audio Datasets

Label noise in audio is rarely just a simple case of the wrong category. It manifests in several distinct ways:

  • Transcription Artifacts: In unsupervised speech recognition, models are trained on transcripts generated by other models. Adrien Dufraux et al. note that standard objectives treat these provided labels as absolute ground truth, leading models to replicate substitutions, insertions, and deletions found in the low-quality source.
  • Temporal Imprecision: Many datasets use "weak labels," where a tag is applied to an entire audio clip regardless of when the sound occurs. Zeynab Raeesy et al. highlight how this results in noise where non-speech or silent frames are incorrectly labeled as the target class, confusing the model's temporal boundaries.
  • Subjectivity and Granularity: In tasks like speech quality assessment, human labels (like MOS scores) are inherently subjective. Junyong Hao et al. argue that using these noisy subjective scores as direct targets hinders the development of robust systems. Similarly, Junyan Jiang et al. show that in music, transcribers often overlook complex chords, penalizing models that are actually more accurate than the ground truth.

Strategies for Robustness

To overcome this sensitivity, researchers have developed mechanisms to "filter" or "bypass" the noise during the training process.

1. Confidence-Based Filtering The most straightforward approach is to prevent the model from seeing high-noise data. Dong Yu et al. propose re-calibrating confidence scores to ensure that only the most reliable machine-generated labels are used for discriminative updates.

2. Latent Variable Modeling Rather than treating a noisy label as a fixed target, some methods treat the "true" label as a hidden variable. The Lead2Gold framework by Adrien Dufraux et al. uses a noise model combined with a differentiable beam search to effectively "clean" the labels during training, allowing the model to look past transcription errors.

3. Wildcard and Uncertainty Tokens New training criteria allow models to explicitly ignore parts of the transcript they find suspicious. Dongji Gao et al. introduced Bypass Temporal Classification (BTC), which uses a "wildcard" token. This gives the model the flexibility to skip over substituted or extra words in a noisy transcript, maintaining high accuracy even when the majority of training labels are incorrect.

4. Better Initializations Starting the training process from a more informed state can reduce the impact of later noise. Ngoc Thang Vu et al. show that initializing models with multilingual data provides a robust feature foundation, which is particularly critical in zero-resource scenarios where the only available supervision is inherently noisy.

By acknowledging that discriminative objectives are "brittle" in the face of error, the field has moved toward training regimes that prioritize acoustic fidelity over strict label adherence. The goal is no longer just to match the label, but to understand the signal well enough to know when the label is wrong.

Key Foundational Work

Unsupervised training with directed manual transcription for recognising Mandarin broadcast audio (2007) — contributed 4 concept elements to this direction.
Initialization Schemes for Multilayer Perceptron Training and their Impact on ASR Performance using Multilingual Data (2012) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_discriminative_training_sensitivity_f164cec7,
  title = {discriminative training sensitivity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/discriminative-training-sensitivity/rdr_f164cec7452c4425b1da497a2fca6dcd},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_discriminative_training_sensitivity_f164cec7_v1,
  title = {discriminative training sensitivity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/discriminative-training-sensitivity/rdr_f164cec7452c4425b1da497a2fca6dcd/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake