TL;DR
Researchers have developed a more precise way to scrub "ums" and "uhs" from audio by treating the most confusing non-filler words (like "a" or "the") as their own distinct categories during training. By combining this "hard category mining" with a specialized inter-category focal loss, their system achieves a state-of-the-art F1 score of 81.0 on the PodcastFillers dataset, significantly outperforming previous ASR-based and ASR-free methods.
What Problem Does This Paper Address?
In spontaneous speech, filler words—disfluencies like "um," "uh," and "ah"—can make a speaker sound less professional or confident. While we want to automate the removal of these words in podcasts and videos, it is surprisingly difficult for AI to do so accurately.
The core challenge is phonetic similarity. A filler word like "uh" sounds remarkably similar to the word "a," and "um" can be easily confused with "I'm" or the start of "umbrella." Most existing systems use a binary classification: "filler" vs. "everything else." By lumping all non-filler words into a single "others" category, the model fails to learn the subtle acoustic nuances required to distinguish a filler from a legitimate word that sounds nearly identical.
Methods
The authors propose an end-to-end, "ASR-free" (Automatic Speech Recognition-free) method. Instead of transcribing the whole text first, it treats filler word detection as a Keyword Spotting (KWS) task.
The system introduces two novel mechanisms:
- Dynamic Auxiliary Category Selection: Instead of just training on "filler" and "non-filler," the model starts with several "placeholder" categories. Every few thousand iterations, the system analyzes which non-filler words are causing the most false positives (e.g., the model keeps thinking "and" is a filler). It then "promotes" that specific word to its own auxiliary category. This forces the neural network to explicitly learn the features of that specific "hard" word to tell it apart from fillers.
- Inter-Category Focal Loss: Standard focal loss helps models focus on hard-to-classify samples. The authors took this further by designing a loss function that adaptively adjusts the penalty weight specifically between the "filler" and "non-filler" categories. If the model is highly confident in a wrong prediction (e.g., calling a "the" a filler), the penalty is increased to sharpen the decision boundary.
The architecture uses a ResNet backbone to extract features from STFT (Short-Time Fourier Transform) spectrograms, followed by three prediction heads that determine the word's heatmap (location), length, and offset.
Datasets
The model was trained and evaluated on the PodcastFillers dataset, a large-scale benchmark for filler word detection. The authors compared their results against several baselines:
- Gentle: A forced-aligner based method.
- Filler-CRNN: A neural network approach.
- VC-FillerNet: A two-stage ASR-free detection method.
Key Insights and Ideas
- The "Monolith" Problem: The biggest "aha" moment is that treating the "non-filler" class as a monolith is a mistake. By breaking out the "hard" words (like "a," "the," "I," and "[repetitions]") into their own classes, the model's latent space becomes much more organized.
- Dynamic Training: You don't need to know which words are confusing ahead of time. The model "discovers" its own weaknesses during the training loop and adjusts its categories accordingly.
- Visual Proof: t-SNE projections in the paper show that without these techniques, filler and non-filler features are a tangled mess. With them, the "filler" category forms a distinct, tight cluster, making it much easier for the decoder to pluck them out of a stream of speech.
Key Results
The proposed method set a new standard for the PodcastFillers dataset:
- F1 Score: Achieved 81.0, a massive jump from the previous best ASR-free method, VC-FillerNet (71.0).
- Precision vs. Recall: The system maintained a very high balance (80.9 Precision / 81.0 Recall), whereas previous models often sacrificed one for the other.
- Ablation Success: Removing the auxiliary categories dropped the F1 score to 74.9, proving that "mining" those hard words is the most critical part of the system's success.
- Optimal Head Count: The researchers found that using 8 auxiliary categories (targeting words like "and," "a," and "the") provided the best performance balance.
Limitations and Future Work
The authors note that the capacity of the model's backbone is a limiting factor; you cannot add an infinite number of auxiliary categories without eventually overwhelming the network.
While the system is "ASR-free" and thus more computationally efficient than full transcription models, it still requires high-quality labels for the "hard" non-filler words during the mining phase. Future work might explore how to apply these "inter-category" logic to even more complex sound events beyond just filler words, such as laughter or background noise interference.
Summary updated