Enhancing the Robustness of Contextual ASR to Varying Biasing Information Volumes Through Purified Semantic Correlation Joint Modeling

Yue Gu, Zhihao Du, Ying Shi, Shiliang Zhang, Qian Chen, Jiqing Han

TL;DR

Researchers have developed PSC-Joint, a novel approach to contextual Automatic Speech Recognition (ASR) that remains accurate even when the list of "hotwords" (biasing phrases) grows very large. By jointly modeling semantic correlations at the list, phrase, and token levels and using a "divide-and-conquer" purification strategy, the system achieves up to a 28% relative improvement in F1 scores over existing methods.

What Problem Does This Paper Address?

Contextual ASR allows speech recognizers to prioritize specific phrases, such as contact names ("Call Zhihao"), song titles, or niche technical jargon. These are provided to the model as a biasing list.

The industry standard for this is cross-attention, where the model "looks" at the biasing list to find matches. However, cross-attention has a major weakness: volume sensitivity. As the biasing list grows from a few dozen to thousands of phrases, the model gets "confused" by distractors—phrases that sound similar to the target. This leads to two issues:

  1. Under-biasing: Failing to recognize a phrase that is in the list.
  2. Over-biasing: Incorrectly "hallucinating" a biasing phrase when the speaker didn't say it.

Existing solutions often try to train models with more distractors, but this frequently degrades performance on smaller, simpler lists.

Methods

The authors propose Purified Semantic Correlation Joint Modeling (PSC-Joint). The core idea is that at any given moment in a recording, only a tiny fraction of the biasing list is actually relevant.

1. Multi-Level Semantic Correlation

Instead of a single attention pass, the model calculates correlations at three granularities:

  • List-level: A binary check—is any phrase in this list relevant to the current audio segment?
  • Phrase-level: Which specific phrases in the list are likely candidates?
  • Token-level: Which specific characters or word pieces within those phrases match the ASR's internal vocabulary?

By computing the intersection of these three levels, the model "spotlights" the most relevant information and ignores the noise.

2. Group Competitive Purification (GCP)

Jointly modeling three levels of correlation is computationally expensive. To solve this, the authors introduced a "divide-and-conquer" mechanism. The biasing list is split into groups, and phrases within each group "compete" based on their correlation scores. Only the winners move on to the final joint modeling stage. This not only speeds up inference but also physically removes distractors before they can cause confusion.

3. Backbone Integration

The system is built on top of Paraformer, a high-performance non-autoregressive ASR model. It uses a collaborative decoding strategy where the standard ASR output and the "biased" output are interpolated based on the list-level confidence score.

Datasets

The models were evaluated on three major Chinese speech corpora:

  • AISHELL-1: 178 hours of standard Mandarin.
  • KeSpeech: 895 hours of Mandarin and eight sub-dialects (testing real-world generalization).
  • AISHELL-NER: A named entity dataset used to generate realistic biasing lists (Person, Location, Organization) ranging from 51 to nearly 1,200 phrases.

Key Insights and Ideas

  • The "Limited Relevance" Principle: Regardless of list length, only a few phrases are ever relevant to a specific audio frame. PSC-Joint exploits this by filtering early and often.
  • Intersection is Key: Modeling at just the token level is noisy; modeling at just the phrase level is too coarse. The "Aha!" moment is that the consistency across list, phrase, and token levels is a much stronger signal for relevance than any single level alone.
  • Purification as a Performance Booster: Usually, pruning a list is a trade-off for speed. Here, the authors found that purification actually improves accuracy (F1 score) because it removes the "confusion" caused by similar-sounding distractors in a massive list.

Key Results

  • Accuracy Gains: PSC-Joint-P (the purified version) achieved relative F1 score improvements of 21.34% on AISHELL-1 and 28.46% on KeSpeech compared to baseline contextual models.
  • Robustness to Scale: While baseline models saw their performance tank as the biasing list grew from 51 to 1,196 phrases, PSC-Joint remained remarkably stable.
  • Efficiency: The Group Competitive Purification (GCP) achieved a 92.06% retention rate of target phrases while significantly reducing the computational load, allowing the model's inference speed to scale nearly linearly with list length.

Limitations and Future Work

  • Training Assumptions: During the training of the semantic predictor, the authors assumed each sentence contains only one biasing phrase to simplify the classification task. Real-world speech often contains multiple entities.
  • Language Scope: The study focused exclusively on Chinese Mandarin. While the architecture is language-agnostic, its effectiveness on morphologically rich languages (like Finnish) or alphabetic languages (like English) remains to be seen.
  • Future Direction: The authors suggest exploring even larger biasing lists (tens of thousands) and further optimizing the "purification" rounds to handle extremely high-volume industrial applications.

Summary updated

Abstract

Recently, cross-attention-based contextual automatic speech recognition (ASR) models have made notable advancements in recognizing personalized biasing phrases. However, the effectiveness of cross-attention is affected by variations in biasing information volume, especially when the length of the biasing list increases significantly. We find that, regardless of the length of the biasing list, only a limited amount of biasing information is most relevant to a specific ASR intermediate representation. Therefore, by identifying and integrating the most relevant biasing information rather than the entire biasing list, we can alleviate the effects of variations in biasing information volume for contextual ASR. To this end, we propose a purified semantic correlation joint modeling (PSC-Joint) approach. In PSC-Joint, we define and calculate three semantic correlations between the ASR intermediate representations and biasing information from coarse to fine: list-level, phrase-level, and token-level. Then, the three correlations are jointly modeled to produce their intersection, so that the most relevant biasing information across various granularities is highlighted and integrated for contextual recognition. In addition, to reduce the computational cost introduced by the joint modeling of three semantic correlations, we also propose a purification mechanism based on a grouped-and-competitive strategy to filter out irrelevant biasing phrases. Compared with baselines, our PSC-Joint approach achieves average relative F1 score improvements of up to 21.34% on AISHELL-1 and 28.46% on KeSpeech, across biasing lists of varying lengths.
BibTeX
@article{gu2025enhancing,
  author = {Yue Gu and Zhihao Du and Ying Shi and Shiliang Zhang and Qian Chen and Jiqing Han},
  title = {{Enhancing the Robustness of Contextual ASR to Varying Biasing Information Volumes Through Purified Semantic Correlation Joint Modeling}},
  journal = {IEEE Transactions on Audio Speech and Language Processing},
  year = {2025},
  doi = {10.1109/taslpro.2025.3606198},
}
Figure