Z-Scores: A Metric for Linguistically Assessing Disfluency Removal

Maria Teleki, Sai Janjur, Haoran Liu, Oliver Grabner, Ketan Verma, Thomas Docog, Xiangjue Dong, Lingfeng Shi, Cong Wang, Stephanie Birkelbach, Jason Kim, Yin Zhang, James CaverleearXiv 2025 · arXiv · OpenReview

Abstract

Evaluating disfluency removal in speech requires more than aggregate token-level scores. Traditional word-based metrics such as precision, recall, and F1 (E-Scores) capture overall performance but cannot reveal why models succeed or fail. We introduce Z-Scores, a span-level linguistically-grounded evaluation metric that categorizes system behavior across distinct disfluency types (EDITED, INTJ, PRN). Our deterministic alignment module enables robust mapping between generated text and disfluent transcripts, allowing Z-Scores to expose systematic weaknesses that word-level metrics obscure. By providing category-specific diagnostics, Z-Scores enable researchers to identify model failure modes and design targeted interventions -- such as tailored prompts or data augmentation -- yielding measurable performance improvements. A case study with LLMs shows that Z-Scores uncover challenges with INTJ and PRN disfluencies hidden in aggregate F1, directly informing model refinement strategies.

Introduction

Spontaneous speech is inherently messy, characterized by interruptions, hesitations, and self-corrections. These "disfluencies" pose significant challenges for downstream natural language processing tasks, such as machine translation or voice assistant command parsing. Traditionally, disfluency removal has been evaluated using word-level Precision, Recall, and F1 scores—collectively referred to as $\mathcal{E}$-Scores. However, these metrics are "linguistically blind"; they provide a single performance number without distinguishing between different types of speech phenomena.

Examples of disfluency types (INTJ, EDITED, PRN) in common smart device interactions.

As shown in Figure 1, disfluencies typically fall into three categories: Interjections (INTJ, e.g., "uh," "um"), Edited Repairs (EDITED, e.g., "gas station — no, grocery store"), and Parentheticals (PRN, e.g., "you know"). To improve model performance, researchers need diagnostic tools that can pinpoint which of these specific categories a model is failing to handle.

Methodology: Beyond Classification

Modern Large Language Models (LLMs) treat disfluency removal as a generative task rather than a simple token classification problem. This shift introduces a new challenge: alignment. LLM outputs may not perfectly match the original transcript's token sequence, making traditional word-for-word comparison difficult.

Comparison of the proposed Z-Score pipeline with traditional classification-based evaluation methods.

The proposed framework (Figure 5) addresses this through a two-step process:

  1. Alignment Module ($\mathcal{A}$): A deterministic alignment module based on Gestalt matching maps the model's generated text ($t_{\Phi}$) back to the original disfluent transcript ($t_{disfluent}$). This ensures that the evaluation strictly measures what was removed or retained, filtering out potential model "hallucinations."
  2. Z-Scores: Instead of an aggregate score, the framework calculates span-level metrics for three linguistic categories: $Z_E$ (EDITED) for repairs and false starts, $Z_I$ (INTJ) for interjections, and $Z_P$ (PRN) for discourse markers.

Case Study: Diagnosing and Fixing LLM Weaknesses

Using the Switchboard dataset and GPT-4o-mini, the authors demonstrated the diagnostic power of Z-Scores. While the model achieved a respectable aggregate F1 score, Z-Scores revealed a stark performance gap: the model was highly effective at removing Edited repairs ($Z_E = 85.20$) but struggled significantly with Interjections ($Z_I = 61.89$) and Parentheticals ($Z_P = 65.02$).

This diagnostic insight allowed for targeted intervention via "metaprompting." By adding explicit instructions and examples focusing on interjections and parentheticals, the researchers achieved significant performance gains. $Z_I$ scores increased by approximately 16 points and $Z_P$ by 9 points, while $Z_E$ remained stable. This localized improvement confirms that Z-Scores can guide researchers toward specific model refinements that aggregate metrics would obscure.

Conclusion

Z-Scores provide a necessary diagnostic lens for the field of disfluency removal. By moving from token-level aggregate scores to span-level linguistic categories, researchers can better understand model failure modes and design targeted improvements. The researchers have released an open-source Python package to facilitate the adoption of Z-Scores as a new standard for evaluating spontaneous speech processing.

Reference: Teleki, M., et al. (2025). Z-Scores: A Metric for Linguistically Assessing Disfluency Removal. arXiv:2509.20319v1 [cs.CL].

Summary updated

Abstract

Evaluating disfluency removal in speech requires more than aggregate token-level scores. Traditional word-based metrics such as precision, recall, and F1 (E-Scores) capture overall performance but cannot reveal why models succeed or fail. We introduce Z-Scores, a span-level linguistically-grounded evaluation metric that categorizes system behavior across distinct disfluency types (EDITED, INTJ, PRN). Our deterministic alignment module enables robust mapping between generated text and disfluent transcripts, allowing Z-Scores to expose systematic weaknesses that word-level metrics obscure. By providing category-specific diagnostics, Z-Scores enable researchers to identify model failure modes and design targeted interventions -- such as tailored prompts or data augmentation -- yielding measurable performance improvements. A case study with LLMs shows that Z-Scores uncover challenges with INTJ and PRN disfluencies hidden in aggregate F1, directly informing model refinement strategies.
BibTeX
@misc{teleki2025z,
  author = {Maria Teleki and Sai Janjur and Haoran Liu and Oliver Grabner and Ketan Verma and Thomas Docog and Xiangjue Dong and Lingfeng Shi and Cong Wang and Stephanie Birkelbach and Jason Kim and Yin Zhang and James Caverlee},
  title = {{Z-Scores: A Metric for Linguistically Assessing Disfluency Removal}},
  year = {2025},
  eprint = {2509.20319},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.20319},
}
Figure