Statistical Testing on ASR Performance via Blockwise Bootstrap

Zhe Liu, Fuchun Peng

Abstract

A common question being raised in automatic speech recognition (ASR) evaluations is how reliable is an observed word error rate (WER) improvement comparing two ASR systems, where statistical hypothesis testing and confidence interval (CI) can be utilized to tell whether this improvement is real or only due to random chance. The bootstrap resampling method has been popular for such significance analysis which is intuitive and easy to use. However, this method fails in dealing with dependent data, which is prevalent in speech world - for example, ASR performance on utterances from the same speaker could be correlated. In this paper we present blockwise bootstrap approach - by dividing evaluation utterances into nonoverlapping blocks, this method resamples these blocks instead of original data. We show that the resulting variance estimator of absolute WER difference between two ASR systems is consistent under mild conditions. We also demonstrate the validity of blockwise bootstrap method on both synthetic and real-world speech data.

Introduction

In the evaluation of Automatic Speech Recognition (ASR) systems, Word Error Rate (WER) is the primary metric. When System B achieves a lower WER than System A, researchers must determine if this improvement is statistically significant or merely a result of random variation in the test set. Statistical significance testing, particularly the "Ordinary Bootstrap" method, is frequently employed for this purpose. Ordinary bootstrap involves resampling individual sentences with replacement to estimate the variance of the performance metric.

However, a critical flaw in the ordinary bootstrap is the assumption that data points are independent and identically distributed (i.i.d.). In speech processing, utterances are often clustered by speaker, environment, or topic, leading to inherent correlations. For instance, if a system struggles with a specific speaker's accent, all utterances from that speaker are likely to exhibit higher error rates. By treating every sentence as independent, the ordinary bootstrap underestimates the true variance, resulting in overly narrow confidence intervals and potentially leading to "over-optimistic discovery"—claiming significance where none exists.

Methodology: Blockwise Bootstrap

To address the limitations of the ordinary bootstrap, we propose the Blockwise Bootstrap method. This approach accounts for data dependencies by resampling groups of related utterances rather than individual ones.

The process involves four main steps:

  1. Partitioning: The evaluation data is divided into $K$ non-overlapping blocks ($S_1, S_2, \dots, S_K$) based on known metadata, such as speaker ID or conversation ID.
  2. Resampling: Instead of drawing $n$ individual sentences, the algorithm draws $K$ blocks with replacement.
  3. Statistic Calculation: For each bootstrap replicate, the WER difference ($\Delta W$) between the two systems is calculated.
  4. Inference: By repeating this process thousands of times, a distribution of $\Delta W$ is constructed, from which 95% confidence intervals can be derived using percentiles or Gaussian approximations.
Comparison of confidence intervals between ordinary bootstrap (left) and blockwise bootstrap (right).

We provide a mathematical proof demonstrating that this blockwise variance estimator is consistent, meaning it converges to the true variance as data volume increases, even under mild dependency conditions.

Experimental Setup

The validity of the blockwise bootstrap was tested on both synthetic and real-world datasets:

  • Synthetic Data: 3,000 simulated utterances with a fixed ground truth WER difference of -0.5%. Correlation ($\rho$) and block sizes ($d$) were varied to compare the performance of ordinary and blockwise methods.
  • Conversational Speech: An in-house dataset comprising 13,987 utterances across 235 conversations, with blocks defined by conversation ID.
  • AMI Meeting Dataset: A public dataset of 25,741 utterances, where blocks were defined by individual speakers within specific meetings (135 blocks total).

Results and Discussion

The experiments highlight the "independence trap" of ordinary bootstrap. When data contains internal correlations, ordinary bootstrap treats each error as an independent event, whereas blockwise bootstrap recognizes that errors within a block often occur together.

Honest Confidence Intervals

Blockwise bootstrap consistently produces wider confidence intervals than the ordinary method. While wider intervals might seem less precise, they are more accurate representations of the true uncertainty in the data. In simulations with high correlation ($\rho = 0.4$), the ordinary bootstrap's 95% CI only contained the true value 41.2% of the time. In contrast, the blockwise method maintained a near-perfect 95.9% coverage.

Real-World Impact

On the Conversational Speech dataset, blockwise confidence intervals were 1.5 times wider than those from the ordinary bootstrap. This effect was even more pronounced on the AMI dataset, where blockwise intervals were twice as wide.

Histograms of bootstrap replicates for (a) Conversational Speech and (b) AMI Meeting datasets.

The histograms visually demonstrate that the blockwise distribution is much more "spread out," reflecting the higher and more realistic variance captured by the method.

Conclusion

The blockwise bootstrap provides a more reliable framework for ASR significance testing by respecting the inherent dependencies in speech data. Its primary limitation is the requirement for predefined block structures based on metadata. Future research will focus on automated methods to infer these structures directly from data—such as using speaker or text embeddings—when explicit metadata is unavailable.

Summary updated

Abstract

A common question being raised in automatic speech recognition (ASR) evaluations is how reliable is an observed word error rate (WER) improvement comparing two ASR systems, where statistical hypothesis testing and confidence interval (CI) can be utilized to tell whether this improvement is real or only due to random chance. The bootstrap resampling method has been popular for such significance analysis which is intuitive and easy to use. However, this method fails in dealing with dependent data, which is prevalent in speech world - for example, ASR performance on utterances from the same speaker could be correlated. In this paper we present blockwise bootstrap approach - by dividing evaluation utterances into nonoverlapping blocks, this method resamples these blocks instead of original data. We show that the resulting variance estimator of absolute WER difference between two ASR systems is consistent under mild conditions. We also demonstrate the validity of blockwise bootstrap method on both synthetic and real-world speech data.
BibTeX
@misc{liu2019statistical,
  author = {Zhe Liu and Fuchun Peng},
  title = {{Statistical Testing on ASR Performance via Blockwise Bootstrap}},
  year = {2019},
  eprint = {1912.09508},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1912.09508},
}

Loading related content...

Figure