Abstract
Modern Automatic Speech Recognition (ASR) systems based on deep neural networks often function as "black boxes," complicating quality assessment and the establishment of user trust. This research addresses these challenges by providing explanations for ASR transcriptions, defined as the minimal and sufficient subset of audio frames responsible for a given output. We adapt Statistical Fault Localisation (SFL) and Causal theory—techniques originally designed for software testing and image classification—to the ASR domain. Our evaluation across Google API, CMU Sphinx, and Mozilla DeepSpeech models demonstrates that these methods can effectively pinpoint the causal audio segments behind transcriptions.
Introduction: The Black Box Problem
While ASR systems have reached high levels of accuracy, understanding why a model produces a specific transcription remains difficult. Unlike image classification, where a single label is assigned to a fixed-size input, ASR involves variable-length sequences and subjective correctness. A transcription might contain minor errors but remain semantically valid. To bridge this gap, we propose the X-ASR framework, which identifies the specific audio frames that "caused" a transcription, moving toward more accountable and trustworthy speech AI.
The X-ASR Framework
The framework operates through two primary components: transcription labeling and adapted explainable AI (XAI) techniques.
Transcription Labeling
To handle the subjective nature of ASR correctness, we employ similarity metrics to compare original transcriptions with those generated from perturbed (mutated) audio. We utilize:
- Word Error Rate (WER): A literal word-matching metric.
- Sentence BERT: A semantic similarity metric that captures the meaning of the transcription.
Adapted XAI Techniques
We adapt three major techniques to work with temporal audio frames:
- Statistical Fault Localisation (SFL): Ranks audio frames based on their presence in "passing" (semantically similar) versus "failing" (dissimilar) mutants.
- Causal Theory: Identifies the smallest set of "superframes" whose modification would alter the model's output.
- LIME (Baseline): An adapted version of Local Interpretable Model-agnostic Explanations that uses a greedy approach to identify influential audio segments.
Experimental Evaluation
The framework was evaluated using 100 audio samples from the Commonvoice dataset. To ensure generalizability, we tested three distinct ASR architectures: the Google Speech-to-Text API, the CMU Sphinx baseline, and Mozilla DeepSpeech (v0.7.3).
Results and Discussion
Our experiments reveal a critical trade-off between the size of an explanation and its consistency across different models.
Explanation Efficiency
As shown in Figure 1, both SFL and Causal methods generally produce more concise explanations than the LIME baseline. Using Sentence BERT as a similarity metric consistently leads to smaller, more "selective" explanations compared to WER. This suggests that semantic metrics better align with how humans judge speech, focusing on the core message rather than literal word matches.
Consistency Across Models
Consistency measures how similar explanations remain when the underlying ASR model is changed (e.g., from Google to Sphinx). While the Causal method using WER achieved the highest consistency (averaging 0.98), it resulted in significantly larger explanations, often encompassing over 90% of the input frames.
The "sweet spot" was found using SFL with BERT similarity, which achieved a median explanation size of approximately 56% while maintaining a high average consistency of 74% (calculated from the 0.81 and 0.67 values observed in Figure 2).
Conclusion
The X-ASR framework demonstrates that post-hoc, black-box explanation techniques can successfully identify the causal audio segments in ASR systems. By treating audio frames as the fundamental unit of perturbation and leveraging semantic similarity metrics like BERT, we can generate concise and relatively consistent explanations. Future work will explore "white-box" techniques that examine the internal states of neural networks to provide even deeper insights into ASR decision-making processes.
Summary updated