Graph Neural Networks for Contextual ASR With the Tree-Constrained Pointer Generator

Guangzhi Sun, Chao Zhang, Philip C. Woodland

Introduction

Automatic Speech Recognition (ASR) systems often struggle with "long-tail" vocabulary—rare terms, proper nouns, or technical jargon that appear infrequently in training data. To mitigate this, researchers employ contextual biasing, providing the model with a "biasing list" of relevant terms (e.g., contact names or presentation keywords) to improve recognition accuracy.

A state-of-the-art approach for this is the Tree-Constrained Pointer Generator (TCPGen), which organizes biasing words into a prefix tree (trie). During decoding, the model queries the tree to identify valid next subword units. However, standard TCPGen is "myopic": nodes only contain information about their specific subword, lacking context about the full words they belong to. This limitation makes it difficult for the model to determine precisely when to transition to the biasing list.

Methodology: GNN Tree Encodings for Lookahead

To address this "myopia," this paper introduces GNN Tree Encodings to provide the model with "lookahead" capabilities. Instead of a node representing an isolated subword, a Graph Neural Network (GNN) aggregates information from all descendant nodes—effectively the remainder of the word—and propagates it up to the current node. This allows the model to "see" the future branches of a word at each decoding step.

The proposed GNN-based TCPGen pipeline, showing how prefix trees are encoded with GNNs to inform the final probability distribution.

The authors evaluated several GNN architectures for this task:

  • Tree-RNN: A recursive bottom-up encoding approach.
  • Graph Convolutional Network (GCN) & GCNII: Spectral methods that pass messages in parallel across the adjacency matrix.
  • GraphSAGE: A spatial method using max-pooling to aggregate features from neighbors.
  • Hybrid Approaches: Combinations of GCN and GraphSAGE (Additive or Bilinear) were tested to capture both spectral and spatial graph features.

Efficiency is maintained through a parameter-tying scheme, sharing weights across GNN layers to reduce complexity while allowing deep information propagation.

Experimental Setup and Datasets

The system was evaluated on two primary datasets:

  1. LibriSpeech: Large-scale simulations were conducted using biasing lists containing rare words from transcriptions mixed with 1,000 "distractor" words to mimic real-world noise.
  2. AMI Meeting Corpus: This "audio-visual" setup tested the system's robustness in dynamic environments. Biasing lists were automatically extracted via OCR from presentation slides used during meetings.
The contextual biasing workflow for the AMI dataset, illustrating the extraction of biasing lists from presentation slides via OCR.

Key Results and Insights

The integration of GNN lookahead significantly improved performance across all metrics:

  • Massive Error Reduction: On LibriSpeech, the GNN-enhanced TCPGen achieved a 60% reduction in Rare Word Error Rate (R-WER) compared to a standard ASR system and a 30-40% reduction over the original TCPGen.
  • Lookahead Efficacy: By encoding the "future" of a word, the pointer generator can trigger more decisively at the first syllable of a rare word. This is particularly effective for words consisting of 3 to 4 subword pieces.
  • Architectural Synergy: GCNs excel at near-future information, while GraphSAGE preserves salient features from deeper in the tree. Their combination yielded the best overall results.
  • Real-World Application: In the AMI experiments, the system achieved a 30-35% relative R-WER reduction using OCR-derived lists, demonstrating its utility for automatically generated context.

Limitations and Future Directions

Despite these gains, performance begins to degrade as biasing lists scale toward 5,000 words due to the quadratic growth of lookahead nodes. Additionally, constructing the adjacency matrix for unique lists introduces minor latency. Future research will explore more complex graph structures beyond prefix trees and optimization techniques for extremely large-scale biasing (tens of thousands of words).

Summary updated

Abstract

Incorporating biasing words obtained through contextual knowledge is paramount in automatic speech recognition (ASR) applications. This paper proposes an innovative method for achieving end-to-end contextual ASR using graph neural network (GNN) encodings based on the tree-constrained pointer generator method. GNN node encodings facilitate lookahead for future word pieces in the process of ASR decoding at each tree node by incorporating information about all word pieces on the tree branches rooted from it. This results in a more precise prediction of the generation probability of the biasing words. The study explores three GNN encoding techniques: namely the tree recursive neural network (Tree-RNN), the graph convolutional network (GCN), and GraphSAGE, along with different combinations of the complementary GCN and GraphSAGE structures. The performance of the systems was evaluated using both Librispeech and the AMI corpus with a visual-grounded contextual ASR pipeline. The findings indicate that using GNN encodings achieved consistent and significant reductions in word error rate (WER), particularly for words that are rare or have not been seen during the training process. Notably, on LibriSpeech test sets, the combined GNN proposed in this paper achieved a 20% relative rare word error rate reduction compared to Tree-RNN, 30%–40% compared to standard TCPGen and 60% compared to standard ASR systems without TCPGen.
BibTeX
@article{sun2024graph,
  author = {Guangzhi Sun and Chao Zhang and Philip C. Woodland},
  title = {{Graph Neural Networks for Contextual ASR With the Tree-Constrained Pointer Generator}},
  journal = {IEEE ACM Trans. Audio Speech Lang. Process.},
  year = {2024},
  doi = {10.1109/taslp.2024.3389645},
}
Figure