Abstract
Evaluating disfluency removal in speech requires more than aggregate token-level scores. Traditional word-based metrics such as precision, recall, and F1 (E-Scores) capture overall performance but cannot reveal why models succeed or fail. We introduce Z-Scores, a span-level linguistically-grounded evaluation metric that categorizes system behavior across distinct disfluency types (EDITED, INTJ, PRN). Our deterministic alignment module enables robust mapping between generated text and disfluent transcripts, allowing Z-Scores to expose systematic weaknesses that word-level metrics obscure. By providing category-specific diagnostics, Z-Scores enable researchers to identify model failure modes and design targeted interventions -- such as tailored prompts or data augmentation -- yielding measurable performance improvements. A case study with LLMs shows that Z-Scores uncover challenges with INTJ and PRN disfluencies hidden in aggregate F1, directly informing model refinement strategies.
Introduction
Spontaneous speech is inherently messy, characterized by interruptions, hesitations, and self-corrections. These "disfluencies" pose significant challenges for downstream natural language processing tasks, such as machine translation or voice assistant command parsing. Traditionally, disfluency removal has been evaluated using word-level Precision, Recall, and F1 scores—collectively referred to as $\mathcal{E}$-Scores. However, these metrics are "linguistically blind"; they provide a single performance number without distinguishing between different types of speech phenomena.
As shown in Figure 1, disfluencies typically fall into three categories: Interjections (INTJ, e.g., "uh," "um"), Edited Repairs (EDITED, e.g., "gas station — no, grocery store"), and Parentheticals (PRN, e.g., "you know"). To improve model performance, researchers need diagnostic tools that can pinpoint which of these specific categories a model is failing to handle.
Methodology: Beyond Classification
Modern Large Language Models (LLMs) treat disfluency removal as a generative task rather than a simple token classification problem. This shift introduces a new challenge: alignment. LLM outputs may not perfectly match the original transcript's token sequence, making traditional word-for-word comparison difficult.
The proposed framework (Figure 5) addresses this through a two-step process:
- Alignment Module ($\mathcal{A}$): A deterministic alignment module based on Gestalt matching maps the model's generated text ($t_{\Phi}$) back to the original disfluent transcript ($t_{disfluent}$). This ensures that the evaluation strictly measures what was removed or retained, filtering out potential model "hallucinations."
- Z-Scores: Instead of an aggregate score, the framework calculates span-level metrics for three linguistic categories: $Z_E$ (EDITED) for repairs and false starts, $Z_I$ (INTJ) for interjections, and $Z_P$ (PRN) for discourse markers.
Case Study: Diagnosing and Fixing LLM Weaknesses
Using the Switchboard dataset and GPT-4o-mini, the authors demonstrated the diagnostic power of Z-Scores. While the model achieved a respectable aggregate F1 score, Z-Scores revealed a stark performance gap: the model was highly effective at removing Edited repairs ($Z_E = 85.20$) but struggled significantly with Interjections ($Z_I = 61.89$) and Parentheticals ($Z_P = 65.02$).
This diagnostic insight allowed for targeted intervention via "metaprompting." By adding explicit instructions and examples focusing on interjections and parentheticals, the researchers achieved significant performance gains. $Z_I$ scores increased by approximately 16 points and $Z_P$ by 9 points, while $Z_E$ remained stable. This localized improvement confirms that Z-Scores can guide researchers toward specific model refinements that aggregate metrics would obscure.
Conclusion
Z-Scores provide a necessary diagnostic lens for the field of disfluency removal. By moving from token-level aggregate scores to span-level linguistic categories, researchers can better understand model failure modes and design targeted improvements. The researchers have released an open-source Python package to facilitate the adoption of Z-Scores as a new standard for evaluating spontaneous speech processing.
Reference: Teleki, M., et al. (2025). Z-Scores: A Metric for Linguistically Assessing Disfluency Removal. arXiv:2509.20319v1 [cs.CL].
Summary updated