Filler Word Detection with Hard Category Mining and Inter-Category Focal Loss

Zhiyuan Zhao, Lijun Wu, Chuanxin Tang, Dacheng Yin, Yucheng Zhao, Chong Luo

Abstract

Filler words like "um" or "uh" are common in spontaneous speech. It is desirable to automatically detect and remove them in recordings, as they affect the fluency, confidence, and professionalism of speech. Previous studies and our preliminary experiments reveal that the biggest challenge in filler word detection is that fillers can be easily confused with other hard categories like "a" or "I". In this paper, we propose a novel filler word detection method that effectively addresses this challenge by adding auxiliary categories dynamically and applying an additional inter-category focal loss. The auxiliary categories force the model to explicitly model the confusing words by mining hard categories. In addition, inter-category focal loss adaptively adjusts the penalty weight between "filler" and "non-filler" categories to deal with other confusing words left in the "non-filler" category. Our system achieves the best results, with a huge improvement compared to other methods on the PodcastFillers dataset.

Introduction

In spontaneous speech, filler words—disfluencies like "um," "uh," and "ah"—can diminish a speaker's perceived professionalism and confidence. Automating the removal of these words in podcasts and videos is a significant challenge due to phonetic similarity. For instance, a filler word like "uh" sounds remarkably similar to the word "a," and "um" can be easily confused with "I'm" or the start of "umbrella." Traditional binary classification systems ("filler" vs. "non-filler") often fail to capture the subtle acoustic nuances required to distinguish fillers from legitimate, phonetically similar words.

Proposed Method

The proposed system treats filler word detection as a Keyword Spotting (KWS) task using an end-to-end, ASR-free approach. The architecture, shown in Figure 1, utilizes a ResNet backbone to extract features from STFT (Short-Time Fourier Transform) spectrograms, followed by prediction heads for word heatmaps, length, and offset.

Proposed system architecture featuring hard class mining and inter-category focal loss.

Two novel mechanisms are introduced to address phonetic confusion:

  1. Dynamic Auxiliary Category Selection: The model dynamically "promotes" non-filler words that cause frequent false positives (e.g., "and," "a," "the") into their own auxiliary categories. This forces the network to explicitly learn features that distinguish these "hard" words from fillers.
  2. Inter-Category Focal Loss: This loss function adaptively adjusts penalty weights between "filler" and "non-filler" categories. By increasing the penalty for high-confidence incorrect predictions, the model develops a sharper decision boundary.

Experimental Results

The model was evaluated on the PodcastFillers dataset, a large-scale benchmark for filler word detection. The system achieved a state-of-the-art F1 score of 81.0, significantly outperforming previous ASR-free methods like VC-FillerNet (71.0).

Performance comparison and ablation study on the PodcastFillers dataset.

Ablation studies demonstrate the importance of the proposed components. Removing auxiliary categories resulted in a drop in F1 score to 74.9, highlighting the critical role of hard category mining. The researchers found that using 8 auxiliary categories provided the optimal balance of performance.

Conclusion

By moving away from a monolithic "non-filler" category and implementing inter-category focal loss, this method significantly improves the precision and recall of filler word detection. While the system requires high-quality labels for the mining phase, it offers a computationally efficient alternative to full ASR-based transcription models. Future work will explore applying these techniques to other complex sound events like laughter or environmental noise.

Summary updated

Abstract

Filler words like ``um" or ``uh" are common in spontaneous speech. It is desirable to automatically detect and remove them in recordings, as they affect the fluency, confidence, and professionalism of speech. Previous studies and our preliminary experiments reveal that the biggest challenge in filler word detection is that fillers can be easily confused with other hard categories like ``a" or ``I". In this paper, we propose a novel filler word detection method that effectively addresses this challenge by adding auxiliary categories dynamically and applying an additional inter-category focal loss. The auxiliary categories force the model to explicitly model the confusing words by mining hard categories. In addition, inter-category focal loss adaptively adjusts the penalty weight between ``filler" and ``non-filler" categories to deal with other confusing words left in the ``non-filler" category. Our system achieves the best results, with a huge improvement compared to other methods on the PodcastFillers dataset.
BibTeX
@inproceedings{zhao2023filler,
  author = {Zhiyuan Zhao and Lijun Wu and Chuanxin Tang and Dacheng Yin and Yucheng Zhao and Chong Luo},
  title = {{Filler Word Detection with Hard Category Mining and Inter-Category Focal Loss}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10095953},
}
Figure