STEAMROLLER: A Multi-Agent System for Inclusive Automatic Speech Recognition for People Who Stutter

Ziqi Xu, Yi Liu, Yuekang Li, Ling Shi, Kailong Wang, Yi Zhao

TL;DR

STEAMROLLER is a real-time AI pipeline designed to make voice technology inclusive for People Who Stutter (PWS) by transforming disfluent speech into fluent, clear audio. By using a novel multi-agent LLM framework to "repair" messy transcriptions and zero-shot voice cloning to maintain the speaker's identity, the system significantly reduces word error rates and improves communication accessibility.

What Problem Does This Paper Address?

In an increasingly voice-driven world—dominated by voice assistants, automated phone menus, and remote meetings—People Who Stutter (PWS) are being systematically left behind. Current Automatic Speech Recognition (ASR) systems from tech giants are trained almost exclusively on fluent speech. When a PWS uses these systems, the ASR often misinterprets repetitions, prolongations, or blocks as entirely different words, leading to "semantic distortion" (the meaning is lost) or total system failure.

The authors identify three core technical hurdles:

  1. Direct Conversion Difficulty: Converting stuttered audio directly to fluent audio is technically brittle with current models.
  2. Semantic Loss: Messy transcriptions often lose the speaker's original intent.
  3. Latency: For a tool to be useful in conversation, it must process speech in near real-time.

Methods

STEAMROLLER moves away from "end-to-end" audio models, opting instead for a three-stage "Divide-and-Conquer" pipeline:

  1. Speech-to-Text (ASR): The system uses a standard model (like OpenAI’s Whisper) to generate an initial transcript. This transcript is often full of errors (e.g., "pa- pa- passenger").
  2. Multi-Agent Text Repair (The Core Innovation): This is where the "STEAMROLLER" magic happens. Instead of a single LLM pass, the system employs a cluster of AI agents:
  3. Repair Agents: Multiple agents (powered by GPT-4o) independently attempt to fix the transcript using Chain-of-Thought reasoning to identify and remove five specific types of stuttering.
  4. Master Agent: This agent acts as an editor. It evaluates the repairs for stability and coherence. If the repairs aren't consistent, it provides feedback to the Repair Agents for up to three iterative refinements.
  5. Text-to-Speech (Voice Cloning): To ensure the output still sounds like the user, the system uses StyleTTS2, a zero-shot diffusion-based voice cloning model. It captures the user's pitch and timbre from the original audio and applies it to the newly repaired, fluent text.

Datasets

The researchers evaluated the system using several key benchmarks:

  • FluencyBank: An English dataset containing 25 hours of stuttered speech from 48 speakers.
  • AS-70: A Mandarin stuttered speech dataset (70 hours) used to prove the system's cross-lingual adaptability.
  • SEP-28K: A dataset used to demonstrate how STEAMROLLER can be used to automatically annotate and improve existing stuttering data.

Key Insights and Ideas

  • Text as a Controllable Bridge: By converting audio to text before "repairing" it, the authors gain much higher controllability over the final output compared to trying to "fix" the raw audio waves.
  • Collaborative Reflection: The multi-agent approach mimics a human editorial process. One agent might over-correct, while another might miss a disfluency; the Master Agent ensures a consensus is reached, leading to much higher "semantic fidelity."
  • The "Inclusive AI Ecosystem" Loop: The authors discovered that by using STEAMROLLER to "clean" stuttered audio and then fine-tuning standard ASR models on that cleaned data, the base ASR models actually got better at recognizing stuttered speech. This suggests STEAMROLLER isn't just an assistive tool, but a engine for building better AI.

Key Results

  • Error Reduction: Across various ASR models (Whisper, wav2vec2, etc.), STEAMROLLER reduced the average Word Error Rate (WER) from 31.60% to 18.69%.
  • Semantic Accuracy: Semantic similarity scores (how well the meaning was preserved) jumped from 0.81 to 0.91.
  • User Satisfaction: In a study of 23 PWS, participants gave the system a high satisfaction score of 4.30/5. They specifically noted that the system helped them "express themselves more fluently and be better understood."
  • Real-Time Performance: The system maintained a latency of approximately 3.77 seconds for 50-100 word segments, making it viable for many professional and social contexts.

Limitations and Future Work

  • Voice "Warmth": While the voice cloning captures the speaker's identity, some users felt the synthesized voice lacked "warmth" or the exact emotional nuance of the original.
  • Paralinguistic Loss: By removing all fillers and repetitions, the system might accidentally strip away intentional emphasis or hesitation that carries social meaning.
  • Spontaneous Conversation: While 3.7 seconds is fast, it can still feel like a "lag" in a very fast-paced, spontaneous back-and-forth conversation.
  • Future Directions: The team aims to refine the AI's ability to distinguish between "intentional" repetition and stuttering-induced repetition to better preserve the speaker's stylistic choices.

Summary updated

Abstract

People who stutter (PWS) face systemic exclusion in today’s voice-driven society, where access to voice assistants, authentication systems, and remote work tools increasingly depends on fluent speech. Current automatic speech recognition (ASR) systems, trained predominantly on fluent speech, fail to serve millions of PWS worldwide. We present STEAMROLLER, a real time system that transforms stuttered speech into fluent output through a novel multi-stage, multi-agent AI pipeline. Our approach addresses three critical technical challenges: (1) the difficulty of direct speech to speech conversion for disfluent input, (2) semantic distortions introduced during ASR transcription of stuttered speech, and (3) latency constraints for real time communication. STEAMROLLER employs a three stage architecture comprising ASR transcription, multi-agent text repair, and speech synthesis, where our core innovation lies in a collaborative multi-agent framework that iteratively refines transcripts while preserving semantic intent. Experiments on the FluencyBank dataset and a user study demonstrates clear word error rate (WER) reduction and strong user satisfaction. Beyond immediate accessibility benefits, fine tuning ASR on STEAMROLLER repaired speech further yields additional WER improvements, creating a pathway toward inclusive AI ecosystems.
BibTeX
@inproceedings{xu2026steamroller,
  author = {Ziqi Xu and Yi Liu and Yuekang Li and Ling Shi and Kailong Wang and Yi Zhao},
  title = {{STEAMROLLER: A Multi-Agent System for Inclusive Automatic Speech Recognition for People Who Stutter}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  doi = {10.1609/aaai.v40i46.41300},
}
Figure