This survey paper, "Fillers in Spoken Language Understanding: Computational and Psycholinguistic Perspectives," provides a comprehensive look at the "uhs" and "ums" of human speech. While often dismissed as mere noise, the authors argue that these fillers are rich with information about a speaker’s cognitive state and communicative intent, and that modern Spoken Language Understanding (SLU) systems ignore them at their own peril.
TL;DR
This paper bridges the gap between psycholinguistics and computer science to show that fillers ("uh", "um") are not just speech errors, but vital communicative signals. It surveys how these events are produced by humans, how they should be handled by AI—moving from simple removal to sophisticated integration—and the ethical trade-offs of synthesizing them in conversational agents.
What Problem Does This Paper Address?
Spontaneous speech is messy. Roughly 5–10% of natural human-to-human conversation consists of disfluencies—interruptions like silent pauses, repetitions, and fillers. Fillers are the most frequent of these events.
Historically, the fields of Spoken Language Understanding (SLU) and Automatic Speech Recognition (ASR) have treated fillers as "noisy channel" interference. The goal was simple: detect them and delete them to produce a "clean" transcript that a machine can easily parse. However, this approach discards significant social and cognitive context. The authors address the lack of a unified resource that connects why humans use fillers (the psycholinguistic view) with how machines can leverage them (the computational view).
Methods
As a survey article, the paper synthesizes a breadth of interdisciplinary research rather than proposing a single new algorithm. The authors categorize the research into two primary lenses:
- Psycholinguistic Perspective: Examining how fillers relate to Production (e.g., cognitive load, strategic choice to hold the floor) and Comprehension (e.g., how listeners use fillers as cues to predict upcoming difficult words).
- Computational Perspective: Contrasting the traditional Removal approach (used in Spoken Dialogue Systems to avoid semantic confusion) with the emerging Integration approach (using fillers as features for social computing, such as detecting emotion, confidence, or even early signs of dementia).
The authors provide a helpful "legend" to navigate these complementary perspectives:
Datasets
The paper discusses several key resources used in the field:
- Switchboard (SWBD): The gold standard corpus for disfluency detection research, consisting of telephone conversations.
- Zhu et al. (2022) Benchmark: A recently released dataset based on podcast episodes, designed specifically for filler word detection and classification in diverse speaking styles.
- Medical ASR Models: The authors note that while general ASR often maps fillers to silence, specialized models (like Google’s medical ASR) have begun including them to capture clinical nuances.
Key Insights and Ideas
- Fillers as "Vocal Gestures": Psycholinguistic evidence suggests fillers are often intentional. They can be "time-buying measures" that signal to a listener that the speaker is still thinking and hasn't finished their turn.
- The "Aha" Moment for Language Models: Research shows that including fillers in language models can actually reduce uncertainty (perplexity). A filler like "uh" often signals that a low-frequency or "hard" word is coming next, helping the model narrow down possibilities.
- The Removal Paradox: In standard Spoken Dialogue Systems (SDS), fillers are removed to prevent "slot-filling" errors (see Figure 1). However, by removing them, the system loses the ability to gauge the user's certainty or emotional state.
- BERT’s Limitation: While modern transformers like BERT have internal representations for fillers, they currently struggle to distinguish between the different functions of "uh" vs. "um," which humans use to signal different lengths of expected delay.
Key Results
- Human Memory: Studies (e.g., Fraundorf & Watson, 2011) found that listeners actually remember information better when it is preceded by a filler compared to a cough or a silent pause of equal duration.
- Social Signaling: In "Broader SLU" tasks, fillers have been successfully used as features to predict personality traits (the "Big 5"), perceived persuasiveness, and speaker roles (e.g., interviewer vs. interviewee).
- TTS Naturalness: Adding fillers to Text-to-Speech (TTS) makes agents sound more human-like, but it can also make them seem less confident.
Limitations and Future Work
The authors highlight several critical challenges for the next generation of SLU:
- The Safety vs. Naturalness Trade-off: If an AI sounds too human by using fillers (like Google Duplex), it may lead to the "Impostor Effect," where users attribute human-level reasoning to the system. This is dangerous in safety-critical contexts like medical advice.
- Lack of Context: Most current computational models treat fillers as isolated tokens. Future work needs to integrate fine-grained context—understanding why a specific filler was used in a specific social situation.
- Robustness "In-the-Wild": Most disfluency detection systems are trained on clean, manually transcribed data. They often break down when faced with the messy, lowercase, unpunctuated output of real-world ASR systems.
Summary updated