In machine learning, the mantra "more data is better" often ignores a critical reality: not all data points are created equal. In speech and language processing, training sets are frequently contaminated by "abnormal" data—mis-transcriptions, background noise, or automated alignment errors. These outliers do not just add noise; they actively skew statistical models, leading to poor generalization and artifacts in generated output.
Research across speaker verification, speech synthesis, and text simplification suggests that identifying and removing a small fraction of these problematic samples—often as little as 2% to 5%—can disproportionately improve model performance.
The Statistical Cost of Abnormal Data
The primary reason outliers degrade performance is their impact on statistical estimation. In speaker verification, for instance, Junhui Liu et al. argue that abnormal data points skew the decision thresholds used to distinguish between genuine users and impostors. By pruning these points, the resulting statistical model becomes more representative of a speaker's true voice characteristics, improving accuracy without requiring new data collection.
This sensitivity is particularly acute in acoustic modeling. Shigeki Matsuda et al. demonstrate that models trained with robust outlier removal can achieve performance levels comparable to models trained on perfectly clean data, even when the original source is riddled with mis-transcriptions. The mechanism works by preventing "bad" data from pulling the model's parameters away from the high-density regions of the true data distribution.
Filtering by Statistical Deviation
A common strategy for identifying these outliers is to look for values that fall outside expected ranges. In speech synthesis, John Kominek shows that removing audio units with extreme durations—those that are unusually long or short—eliminates the artifacts that cause synthetic speech to sound unnatural. Similarly, Yang Wang et al. (2015) utilize percentile-based thresholding to ensure that regression models for speech rhythm are not biased by the noise inherent in automated labeling tools.
This logic extends to natural language tasks. In text simplification, Laura Vásquez-Rodríguez et al. found that removing the top 2% to 5% of sentence pairs with the highest edit distance significantly enhances model robustness. These high-edit-distance pairs often represent noisy or non-representative simplifications that confuse the sequence-to-sequence learning process.
Refining Automated Alignments
As researchers move toward larger, automatically generated datasets, the need for pre-classification filtering grows. In the creation of the BibleTTS corpus, David Ifeoluwa Adelani et al. noted that automated alignment produces noisy audio-text pairs. Pruning these samples is essential for subjective naturalness, even if it results in a smaller final dataset.
Similarly, Yang Wang et al. (2019) highlight that end-to-end synthesis models struggle with abrupt segmentations or background noise in training data. These outliers can cause the model to fail during the alignment phase, leading to synthesis errors like trailing silence or robotic artifacts.
Focusing on Typicality and Discriminative Power
Rather than just removing "bad" data, some researchers focus on selecting "typical" data. Mohamed Mihoubi et al. propose training speaker identification models exclusively on typical speech segments—those that carry the most discriminative information—while discarding atypical segments like silence or non-speech sounds.
This concept of "garbage" removal is also effective in dialect classification. Rongqing Huang et al. identify and remove confusing speech frames that might mislead a classifier. By training only on the most discriminative segments, the model becomes more robust to the variations found in real-world audio.
Iterative and Model-Based Rejection
Advanced filtering methods use the models themselves to identify outliers. Joshua Winebarger et al. describe an iterative data cleaning process where a high-performing model identifies the most difficult training utterances for removal. By purging these problematic samples, the final acoustic model can focus on high-quality signals.
In the context of speaker enrollment, Javier R. Saeta et al. found that replacing outlier utterances with new, more representative data yields better results than simple removal. This suggests that while pruning is powerful, the ultimate goal is to ensure the model's training distribution is as clean and representative as possible.
Key Foundational Work
Pruning abnormal data for better making a decision in speaker verification (2000) — contributed 3 concept elements to this direction.
Mode preference in a simple data-retrieval task (1993) — contributed 1 concept elements to this direction.