Evaluating Speech Recognition Performance Towards Large Language Model Based Voice Assistants

Zhe Liu, Suyoun Kim, Ozlem Kalinli2024 · DOI

TL;DR

This paper introduces new metrics for evaluating Automatic Speech Recognition (ASR) systems specifically for Large Language Model (LLM)-powered voice assistants. By leveraging the LLM itself to embed ASR transcriptions and measure semantic similarity, these new metrics better capture how ASR errors impact the assistant's overall performance compared to traditional metrics like Word Error Rate (WER).

What Problem Does This Paper Address?

LLM-powered voice assistants are becoming increasingly sophisticated, but their performance relies heavily on the accuracy of the initial ASR system that transcribes spoken queries. Traditional ASR evaluation metrics, like Word Error Rate (WER), are often insufficient because they treat all errors equally. For instance, a minor grammatical change might not affect the LLM's understanding, but WER would penalize it. Conversely, a semantically critical error might be missed by WER but lead to a failed task for the voice assistant. This paper addresses the need for ASR evaluation metrics that align with how LLM-based voice assistants perceive and judge the quality of ASR output, ensuring that ASR improvements translate to better user experiences.

Methods

The core of this paper is the proposal of three novel LLM-based semantic distance metrics. The general idea is to use the same LLM that powers the voice assistant to generate semantic embeddings for both the ASR hypothesis and the ground truth reference. The semantic distance is then calculated as the cosine distance between these embeddings.

The three proposed methods differ in how they extract these embeddings:

  1. LLMSemDist-Raw: This method directly prompts the LLM with the raw ASR hypothesis (or reference) and uses the hidden state of the final token from the last Transformer layer as the sentence embedding.
  2. LLMSemDist-Prompt: This approach uses a system prompt template (similar to what's used for the voice assistant itself) with the ASR hypothesis plugged in. The embedding is again derived from the final token's hidden state. This aims to capture the semantic representation as perceived by the assistant.
  3. LLMSemDist-EOWL (Explicit One Word Limitation): This method uses a prompt that instructs the LLM to summarize the input sentence in "one word." Instead of using hidden states, it leverages the logits of the LLM's output for the next token, arguing that these logits better represent the semantic meaning for summarization tasks.

Additionally, the paper introduces a "coupled bootstrap" method for statistically comparing the significance of differences between two ASR systems across both WER and the proposed semantic metrics simultaneously.

Datasets

The experiments were conducted using the public OpenAssistant corpus. A curated subset of approximately 4,000 sentences (under 100 characters) was used to collect speech data via crowdsourcing. This data was anonymized. The ASR model used was an RNN-T model trained on an in-house video dataset. For evaluation, the paper focuses on a subset of ~2.5k utterances where ASR errors occurred and the LLM did not provide useful information even with reference prompts, aiming to isolate the impact of ASR errors on task success.

Key Insights and Ideas

  • LLMs as Semantic Evaluators: The fundamental insight is that the LLM powering the voice assistant is the ultimate judge of ASR quality. Therefore, using that same LLM to evaluate ASR output is a more direct and relevant approach than traditional metrics.
  • Semantic Nuance Matters: Minor ASR errors that don't change the core meaning are often tolerated by LLMs, but WER penalizes them equally to critical errors. Semantic distance metrics can better reflect this tolerance.
  • Prompting Style Influences Embeddings: Different prompting strategies (raw text vs. assistant prompt vs. EOWL) yield different semantic representations, and the "EOWL" method, focusing on summarization logits, shows particular promise.
  • Statistical Rigor for ASR Comparison: The coupled bootstrap method is crucial for reliably determining if improvements in ASR performance (both transcription quality and semantic impact) are statistically significant, not just due to random chance.

Key Results

  • The proposed LLM-based semantic metrics (LLMSemDist-Raw, LLMSemDist-Prompt, LLMSemDist-EOWL) consistently outperformed traditional metrics like WER, RareWER, and even a BERT-based SemDist in predicting task success for the downstream LLM voice assistant.
  • LLMSemDist-EOWL generally showed the strongest predictive power across various evaluation criteria (AUC, Efron-R², McFadden-R²), indicating it's the most effective metric among those tested.
  • When comparing different aggregation methods for LLMSemDist-Raw, taking the average of final token hidden states over Transformer layers yielded the best results.
  • Linear combinations of metrics, particularly LLMSemDist-EOWL with EER (Entity Error Rate), showed further improvements in predictive ability, suggesting that combining semantic and targeted lexical information can be beneficial.
  • Statistical significance tests confirmed that a larger ASR model (1B parameters) was significantly better than a smaller one (80M parameters) in terms of both WER and LLMSemDist-EOWL.

Limitations and Future Work

  • LLM Tolerance: The paper notes that LLMs can tolerate some ASR errors, and the evaluation focuses on utterances where errors do lead to task failure, implying that the metrics are particularly useful for identifying critical ASR failures rather than minor imperfections.
  • Future Work: The authors plan to explore how these proposed metrics can be used as additional objectives for training ASR systems, aiming to directly optimize for LLM-based voice assistant performance. They also suggest further investigation into different aggregation or pooling strategies for LLM embeddings.

Summary updated

BibTeX
@misc{liu2024evaluating,
  author = {Zhe Liu and Suyoun Kim and Ozlem Kalinli},
  title = {{Evaluating Speech Recognition Performance Towards Large Language Model Based Voice Assistants}},
  year = {2024},
  doi = {10.21437/interspeech.2024-635},
}
Figure