Introduction
Automatic Speech Recognition (ASR) systems do not merely produce a single string of words. Instead, they generate complex internal representations of multiple hypotheses, typically structured as confusion networks (CNs) or word lattices. While standard Bi-directional Recurrent Neural Networks (BiRNNs) excel at estimating confidence for the single "1-best" sequence, they are ill-equipped for the branching, multi-path nature of lattices where a single node may have a variable number of incoming and outgoing arcs.
Downstream applications—such as keyword spotting, speaker adaptation, and semi-supervised learning—require reliable confidence scores for all alternative hypotheses within these graphs. This paper introduces a novel extension of BiRNNs designed specifically to handle these non-linear structures, bridging the gap between sequence models and graph-based ASR outputs.
Methods: BiCNRNN and BiLatRNN
To address the limitations of sequence-only models, the authors propose BiCNRNN (for Confusion Networks) and BiLatRNN (for Lattices). These models propagate hidden states through the graph in a manner inspired by the classic forward-backward algorithm, allowing every arc to capture both past and future context.
Attention-Based State Merging
A primary challenge in graph-based RNNs is merging hidden states from a variable number of incoming arcs at each node. Simple heuristics like taking the mean or maximum of incoming states often lose critical information. To solve this, the authors implement an attention mechanism. This mechanism dynamically weights incoming states based on their history and local confidence statistics, allowing the model to intelligently route the most relevant historical context during propagation.
Training and Target Assignment
Training these models requires "ground truth" labels for every arc in the graph. For Confusion Networks, the authors use standard Levenshtein alignment against a reference sequence. However, because exact Levenshtein alignment is computationally prohibitive for massive lattices, they introduce an approximate loss function based on the time-overlap between lattice arcs and reference words. This approach allows for efficient supervised training on large-scale datasets.
Datasets and Experimental Setup
The models were evaluated using the IARPA Babel Georgian Full Language Pack (FLP), a challenging dataset containing approximately 40 hours of conversational telephone speech. The ASR system used to generate the lattices was a diverse ensemble utilizing multi-lingual bottleneck features, originally developed for the IARPA OpenKWS 2016 competition.
Key Results
The results demonstrate that dedicated neural confidence models are essential. Raw ASR posterior probabilities are often overly optimistic; on lattice arcs, they yielded a dismal Normalized Cross-Entropy (NCE) of -5.0386.
As shown in the Precision-Recall curves, the BiLatRNN significantly improves precision, particularly in high-recall scenarios. This is a game-changer for information retrieval, where searching deep into alternative hypotheses is necessary but prone to false positives.
Quantitatively, the BiLatRNN with attention achieved an NCE of 0.3921 and an AUC of 0.7537, vastly outperforming traditional decision tree baselines (NCE -0.0889, AUC 0.2251). On Confusion Networks, the BiCNRNN achieved an NCE of 0.4970 and an AUC of 0.8365, further validating the approach across different graph structures.
Conclusion and Future Work
By extending BiRNNs to handle lattice structures via attention-based state merging, this work provides a robust framework for confidence estimation in ASR. While the current training relies on an approximate time-overlap heuristic, the performance gains are already substantial. Future research will focus on integrating these improved scores into end-to-end pipelines for keyword spotting and speaker adaptation to measure their direct impact on downstream task accuracy.
Summary updated