Graph Neural Networks for Contextual ASR with the Tree-Constrained Pointer Generator

Guangzhi Sun, Chao Zhang, Philip C. Woodland

Abstract

The incorporation of biasing words obtained through contextual knowledge is of paramount importance in automatic speech recognition (ASR) applications. This paper proposes an innovative method for achieving end-to-end contextual ASR using graph neural network (GNN) encodings based on the tree-constrained pointer generator method. GNN node encodings facilitate lookahead for future word pieces in the process of ASR decoding at each tree node by incorporating information about all word pieces on the tree branches rooted from it. This results in a more precise prediction of the generation probability of the biasing words. The study explores three GNN encoding techniques, namely tree recursive neural networks, graph convolutional network (GCN), and GraphSAGE, along with different combinations of the complementary GCN and GraphSAGE structures. The performance of the systems was evaluated using the Librispeech and AMI corpus, following the visual-grounded contextual ASR pipeline. The findings indicate that using GNN encodings achieved consistent and significant reductions in word error rate (WER), particularly for words that are rare or have not been seen during the training process. Notably, the most effective combination of GNN encodings obtained more than 60% WER reduction for rare and unseen words compared to standard end-to-end systems.

Introduction

Modern end-to-end Automatic Speech Recognition (ASR) systems often struggle with the "long-tail" problem, where rare words—such as specific names, technical jargon, or unique entities—that appear infrequently in training data are misrecognized. Contextual biasing addresses this by providing the model with a dynamic "biasing list" of words likely to appear in a specific context (e.g., a user's contact list or meeting slides).

However, integrating these dynamic lists into static, pre-trained models is challenging. Previous methods often lacked "lookahead" capabilities—the ability for the model to determine, while mid-word, whether the remaining word pieces match the contextual list. This paper introduces a novel approach using Graph Neural Networks (GNNs) to encode these biasing lists, providing the necessary lookahead to improve recognition accuracy.

Methodology

The system builds upon the Tree-Constrained Pointer Generator (TCPGen). In this framework, biasing words are organized into a prefix tree (a trie). As the ASR system decodes speech, TCPGen queries this tree to determine if the current path matches any biasing words.

The overall architecture of the GNN-encoded TCPGen system, showing the transformation from a prefix tree to GNN encodings and its integration with the ASR model.

The key innovation is the use of GNNs to encode the prefix tree. Unlike standard encodings where a node only represents its own word piece, a GNN allows information to flow from the "leaves" (word endings) back to the "root." This creates a lookahead functionality: the representation of a prefix (e.g., "un-") incorporates information about all possible completions (e.g., "-seen" or "-usual") present in the biasing list. The study explores three architectures: Tree-RNNs, Graph Convolutional Networks (GCN), and GraphSAGE. To prevent over-smoothing in deep graphs, a parameter-tying scheme and bilinear combinations were utilized to merge the strengths of GCN and GraphSAGE.

Experimental Setup

The models were evaluated on two major benchmarks:

  1. LibriSpeech: A standard audiobook corpus used to test performance on both full (960-hour) and limited (100-hour) data scenarios.
  2. AMI Meeting Corpus: A challenging real-world dataset. For this corpus, the authors implemented a visual-grounded pipeline where biasing lists were automatically extracted from presentation slides using Optical Character Recognition (OCR).
The visual-grounded contextual ASR pipeline for the AMI meeting corpus, illustrating how OCR outputs from slides are used to generate dynamic biasing lists.

Key Results and Discussion

The GNN-based TCPGen demonstrated significant performance gains across all metrics:

  • Rare Word Error Rate (R-WER) Reduction: On LibriSpeech, the best GNN-based systems achieved over a 60% relative reduction in R-WER compared to standard end-to-end systems. This highlights the model's ability to generalize to out-of-vocabulary (OOV) words when they are provided in the biasing list.
  • Overall Accuracy: The GCN model achieved a 16% relative reduction in general Word Error Rate (WER) on the LibriSpeech test-clean set.
  • Complementarity: GCNs (focusing on broad graph structure) and GraphSAGE (focusing on local salient features) proved complementary; their combination yielded superior results compared to individual models.
  • Real-World Application: The AMI meeting experiments confirmed that the system could successfully utilize "noisy" real-world contextual data derived from OCR to improve recognition of specific entities like project names.

Conclusion

By leveraging GNNs to encode prefix trees, this research provides a robust solution to the lookahead problem in contextual ASR. While the GNN encoding adds some training overhead (2.5x to 3.5x), it significantly improves the confidence and accuracy of the pointer generator during inference. Future work will explore applying these GNN encodings to more complex graph structures and other Spoken Language Understanding (SLU) tasks.

Summary updated

Abstract

The incorporation of biasing words obtained through contextual knowledge is of paramount importance in automatic speech recognition (ASR) applications. This paper proposes an innovative method for achieving end-to-end contextual ASR using graph neural network (GNN) encodings based on the tree-constrained pointer generator method. GNN node encodings facilitate lookahead for future word pieces in the process of ASR decoding at each tree node by incorporating information about all word pieces on the tree branches rooted from it. This results in a more precise prediction of the generation probability of the biasing words. The study explores three GNN encoding techniques, namely tree recursive neural networks, graph convolutional network (GCN), and GraphSAGE, along with different combinations of the complementary GCN and GraphSAGE structures. The performance of the systems was evaluated using the Librispeech and AMI corpus, following the visual-grounded contextual ASR pipeline. The findings indicate that using GNN encodings achieved consistent and significant reductions in word error rate (WER), particularly for words that are rare or have not been seen during the training process. Notably, the most effective combination of GNN encodings obtained more than 60% WER reduction for rare and unseen words compared to standard end-to-end systems.
BibTeX
@article{sun2024graph,
  author = {Guangzhi Sun and Chao Zhang and Philip C. Woodland},
  title = {{Graph Neural Networks for Contextual ASR with the Tree-Constrained Pointer Generator}},
  journal = {IEEE ACM Trans. Audio Speech Lang. Process.},
  year = {2024},
  doi = {10.1109/taslp.2024.3389645},
}
Figure