Fillers in Spoken Language Understanding: Computational and Psycholinguistic Perspectives

Tanvi Dinkar, Chloé Clavel, Ioana VasilescuarXiv 2023 · arXiv

This survey paper, "Fillers in Spoken Language Understanding: Computational and Psycholinguistic Perspectives," provides a comprehensive look at the "uhs" and "ums" of human speech. While often dismissed as mere noise, the authors argue that these fillers are rich with information about a speaker’s cognitive state and communicative intent, and that modern Spoken Language Understanding (SLU) systems ignore them at their own peril.

TL;DR

This paper bridges the gap between psycholinguistics and computer science to show that fillers ("uh", "um") are not just speech errors, but vital communicative signals. It surveys how these events are produced by humans, how they should be handled by AI—moving from simple removal to sophisticated integration—and the ethical trade-offs of synthesizing them in conversational agents.

What Problem Does This Paper Address?

Spontaneous speech is messy. Roughly 5–10% of natural human-to-human conversation consists of disfluencies—interruptions like silent pauses, repetitions, and fillers. Fillers are the most frequent of these events.

Historically, the fields of Spoken Language Understanding (SLU) and Automatic Speech Recognition (ASR) have treated fillers as "noisy channel" interference. The goal was simple: detect them and delete them to produce a "clean" transcript that a machine can easily parse. However, this approach discards significant social and cognitive context. The authors address the lack of a unified resource that connects why humans use fillers (the psycholinguistic view) with how machines can leverage them (the computational view).

Methods

As a survey article, the paper synthesizes a breadth of interdisciplinary research rather than proposing a single new algorithm. The authors categorize the research into two primary lenses:

  1. Psycholinguistic Perspective: Examining how fillers relate to Production (e.g., cognitive load, strategic choice to hold the floor) and Comprehension (e.g., how listeners use fillers as cues to predict upcoming difficult words).
  2. Computational Perspective: Contrasting the traditional Removal approach (used in Spoken Dialogue Systems to avoid semantic confusion) with the emerging Integration approach (using fillers as features for social computing, such as detecting emotion, confidence, or even early signs of dementia).

The authors provide a helpful "legend" to navigate these complementary perspectives:

Table 1: Mapping psycholinguistic concepts to computational tasks.

Datasets

The paper discusses several key resources used in the field:

  • Switchboard (SWBD): The gold standard corpus for disfluency detection research, consisting of telephone conversations.
  • Zhu et al. (2022) Benchmark: A recently released dataset based on podcast episodes, designed specifically for filler word detection and classification in diverse speaking styles.
  • Medical ASR Models: The authors note that while general ASR often maps fillers to silence, specialized models (like Google’s medical ASR) have begun including them to capture clinical nuances.

Key Insights and Ideas

  • Fillers as "Vocal Gestures": Psycholinguistic evidence suggests fillers are often intentional. They can be "time-buying measures" that signal to a listener that the speaker is still thinking and hasn't finished their turn.
  • The "Aha" Moment for Language Models: Research shows that including fillers in language models can actually reduce uncertainty (perplexity). A filler like "uh" often signals that a low-frequency or "hard" word is coming next, helping the model narrow down possibilities.
  • The Removal Paradox: In standard Spoken Dialogue Systems (SDS), fillers are removed to prevent "slot-filling" errors (see Figure 1). However, by removing them, the system loses the ability to gauge the user's certainty or emotional state.
Figure 1: Example of SLU for SDS, where the disfluent part of the utterance is highlighted in red and underlined. As shown, the disfluencies are removed in post-processing using disfluency detection systems, after ASR systems transcribe the input speech. Then, the input is further collapsed into a semantic frame, consisting of intent and slot.
  • BERT’s Limitation: While modern transformers like BERT have internal representations for fillers, they currently struggle to distinguish between the different functions of "uh" vs. "um," which humans use to signal different lengths of expected delay.

Key Results

  • Human Memory: Studies (e.g., Fraundorf & Watson, 2011) found that listeners actually remember information better when it is preceded by a filler compared to a cough or a silent pause of equal duration.
  • Social Signaling: In "Broader SLU" tasks, fillers have been successfully used as features to predict personality traits (the "Big 5"), perceived persuasiveness, and speaker roles (e.g., interviewer vs. interviewee).
  • TTS Naturalness: Adding fillers to Text-to-Speech (TTS) makes agents sound more human-like, but it can also make them seem less confident.

Limitations and Future Work

The authors highlight several critical challenges for the next generation of SLU:

  • The Safety vs. Naturalness Trade-off: If an AI sounds too human by using fillers (like Google Duplex), it may lead to the "Impostor Effect," where users attribute human-level reasoning to the system. This is dangerous in safety-critical contexts like medical advice.
  • Lack of Context: Most current computational models treat fillers as isolated tokens. Future work needs to integrate fine-grained context—understanding why a specific filler was used in a specific social situation.
  • Robustness "In-the-Wild": Most disfluency detection systems are trained on clean, manually transcribed data. They often break down when faced with the messy, lowercase, unpunctuated output of real-world ASR systems.

Summary updated

Abstract

Disfluencies (i.e. interruptions in the regular flow of speech), are ubiquitous to spoken discourse. Fillers ("uh", "um") are disfluencies that occur the most frequently compared to other kinds of disfluencies. Yet, to the best of our knowledge, there isn't a resource that brings together the research perspectives influencing Spoken Language Understanding (SLU) on these speech events. This aim of this article is to survey a breadth of perspectives in a holistic way; i.e. from considering underlying (psycho)linguistic theory, to their annotation and consideration in Automatic Speech Recognition (ASR) and SLU systems, to lastly, their study from a generation standpoint. This article aims to present the perspectives in an approachable way to the SLU and Conversational AI community, and discuss moving forward, what we believe are the trends and challenges in each area.
BibTeX
@misc{dinkar2023fillers,
  author = {Tanvi Dinkar and Chloé Clavel and Ioana Vasilescu},
  title = {{Fillers in Spoken Language Understanding: Computational and Psycholinguistic Perspectives}},
  year = {2023},
  eprint = {2301.10761},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2301.10761},
}
Figure