Explanations for Automatic Speech Recognition

Xiaoliang Wu, Peter Bell, Ajitha Rajan

Abstract

Modern Automatic Speech Recognition (ASR) systems based on deep neural networks often function as "black boxes," complicating quality assessment and the establishment of user trust. This research addresses these challenges by providing explanations for ASR transcriptions, defined as the minimal and sufficient subset of audio frames responsible for a given output. We adapt Statistical Fault Localisation (SFL) and Causal theory—techniques originally designed for software testing and image classification—to the ASR domain. Our evaluation across Google API, CMU Sphinx, and Mozilla DeepSpeech models demonstrates that these methods can effectively pinpoint the causal audio segments behind transcriptions.

Introduction: The Black Box Problem

While ASR systems have reached high levels of accuracy, understanding why a model produces a specific transcription remains difficult. Unlike image classification, where a single label is assigned to a fixed-size input, ASR involves variable-length sequences and subjective correctness. A transcription might contain minor errors but remain semantically valid. To bridge this gap, we propose the X-ASR framework, which identifies the specific audio frames that "caused" a transcription, moving toward more accountable and trustworthy speech AI.

The X-ASR Framework

The framework operates through two primary components: transcription labeling and adapted explainable AI (XAI) techniques.

Transcription Labeling

To handle the subjective nature of ASR correctness, we employ similarity metrics to compare original transcriptions with those generated from perturbed (mutated) audio. We utilize:

  • Word Error Rate (WER): A literal word-matching metric.
  • Sentence BERT: A semantic similarity metric that captures the meaning of the transcription.

Adapted XAI Techniques

We adapt three major techniques to work with temporal audio frames:

  1. Statistical Fault Localisation (SFL): Ranks audio frames based on their presence in "passing" (semantically similar) versus "failing" (dissimilar) mutants.
  2. Causal Theory: Identifies the smallest set of "superframes" whose modification would alter the model's output.
  3. LIME (Baseline): An adapted version of Local Interpretable Model-agnostic Explanations that uses a greedy approach to identify influential audio segments.

Experimental Evaluation

The framework was evaluated using 100 audio samples from the Commonvoice dataset. To ensure generalizability, we tested three distinct ASR architectures: the Google Speech-to-Text API, the CMU Sphinx baseline, and Mozilla DeepSpeech (v0.7.3).

Results and Discussion

Our experiments reveal a critical trade-off between the size of an explanation and its consistency across different models.

Explanation Efficiency

As shown in Figure 1, both SFL and Causal methods generally produce more concise explanations than the LIME baseline. Using Sentence BERT as a similarity metric consistently leads to smaller, more "selective" explanations compared to WER. This suggests that semantic metrics better align with how humans judge speech, focusing on the core message rather than literal word matches.

Figure 1: Comparison of explanation sizes (fraction of audio frames) across different methods and metrics.

Consistency Across Models

Consistency measures how similar explanations remain when the underlying ASR model is changed (e.g., from Google to Sphinx). While the Causal method using WER achieved the highest consistency (averaging 0.98), it resulted in significantly larger explanations, often encompassing over 90% of the input frames.

Figure 2: Consistency scores for explanations between Google and Sphinx ASR models.

The "sweet spot" was found using SFL with BERT similarity, which achieved a median explanation size of approximately 56% while maintaining a high average consistency of 74% (calculated from the 0.81 and 0.67 values observed in Figure 2).

Conclusion

The X-ASR framework demonstrates that post-hoc, black-box explanation techniques can successfully identify the causal audio segments in ASR systems. By treating audio frames as the fundamental unit of perturbation and leveraging semantic similarity metrics like BERT, we can generate concise and relatively consistent explanations. Future work will explore "white-box" techniques that examine the internal states of neural networks to provide even deeper insights into ASR decision-making processes.

Summary updated

Abstract

We address quality assessment for neural network based ASR by providing explanations that help increase our understanding of the system and ultimately help build trust in the system. Compared to simple classification labels, explaining transcriptions is more challenging as judging their correctness is not straightforward and transcriptions as a variable-length sequence is not handled by existing interpretable machine learning models. We provide an explanation for an ASR transcription as a subset of audio frames that is both a minimal and sufficient cause of the transcription. To do this, we adapt existing explainable AI (XAI) techniques from image classification-Statistical Fault Localisation(SFL) and Causal. Additionally, we use an adapted version of Local Interpretable Model-Agnostic Explanations (LIME) for ASR as a baseline in our experiments. We evaluate the quality of the explanations generated by the proposed techniques over three different ASR ,Google API, the baseline model of Sphinx, Deepspeech and 100 audio samples from the Commonvoice dataset.
BibTeX
@inproceedings{wu2023explanations,
  author = {Xiaoliang Wu and Peter Bell and Ajitha Rajan},
  title = {{Explanations for Automatic Speech Recognition}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=qvsSeSoikN},
}
Figure