Open-Domain Name Error Detection using a Multi-Task RNN

Hao Cheng, Hao Fang, Mari OstendorfEMNLP 2015 · OpenReview

Abstract

Out-of-vocabulary name errors in speech recognition create significant problems for downstream language processing, but the fact that they are rare poses challenges for automatic detection, particularly in an open-domain scenario. To address this problem, a multi-task recurrent neural network language model for sentence-level name detection is proposed for use in combination with out-of-vocabulary word detection. The sentence-level model is also effective for leveraging external text data. Experiments show a 26% improvement in name-error detection F-score over a system using n-gram lexical features.

Introduction

Automatic Speech Recognition (ASR) systems frequently encounter "Out-of-Vocabulary" (OOV) words, particularly proper names which are diverse and rare. When an ASR system encounters an unknown name, it often "force-fits" it into known phonetic sequences, leading to nonsensical transcriptions. For instance, the name "Litanfeeth" might be misrecognized as "it leaks on feet." Such errors are detrimental to downstream tasks like machine translation or digital assistants. Detecting these errors is challenging due to data sparsity and domain mismatch, where models trained on specific contexts fail to generalize to others.

Methodology: Multi-Task Recurrent Neural Networks

To overcome these challenges, we propose a Multi-Task Recurrent Neural Network (MT RNN). Unlike standard RNNs that focus solely on language modeling (LM), this architecture utilizes two distinct output layers or "heads" sharing a common hidden representation.

The Multi-Task RNN architecture featuring a shared hidden state h(t) that feeds into both a next-word prediction head o(t) and a sentence-level name indicator y(n).

The primary innovation lies in the multi-task objective. The LM head performs standard next-word prediction, acting as a regularizer. Because large volumes of text are available for language modeling, the RNN learns robust continuous-space embeddings of sentences. This shared representation enhances the name-detection task, even when specific name-error training data is limited. The system employs a factored approach, first predicting the likelihood of a name error at the sentence level and then combining this with word-level OOV detection using a Maximum Entropy classifier.

Datasets and Domain Adaptation

The study utilized the BOLT dataset, which includes ASR hypotheses from three domains: TRANSTAC (military), HADR (humanitarian aid), and General (social/diverse). To address the open-domain challenge, we augmented the training data with a Reddit corpus containing 135,000 sentences with names and 360,000 without. This diverse "clean" text helps the model generalize beyond the limited ASR training data.

Experimental Results

The MT RNN demonstrated significant performance gains over traditional methods. By leveraging the entire sentence representation, the model identifies "long-distance" cues that a name is likely present, which local context (often corrupted by ASR errors) might miss.

F-score performance across TRANSTAC, HADR, and General domains, comparing Bag-of-Words (BOW) baselines with Single-Task (ST) and Multi-Task (MT) RNNs, including the impact of additional Reddit training data.

Key findings include:

  • F-Score Improvement: The MT RNN achieved a 26% relative improvement in name-error detection F-score compared to a baseline using standard n-gram lexical features.
  • Domain Generalization: On the "General" domain—the most diverse and difficult test set—the MT RNN significantly outperformed single-task models, particularly when augmented with Reddit data.
  • Precision Gains: The RNN approach primarily improved precision, reducing the frequency of "false alarms" in error detection.

Conclusion

The proposed MT RNN effectively addresses the sparsity and domain-mismatch issues inherent in open-domain name error detection. By sharing representations between language modeling and name detection, the model leverages large-scale text data to improve performance significantly. Future work will explore more complex architectures like LSTMs or GRUs and investigate tighter integration of sentence representations into word-level error detectors.

Summary updated

Abstract

Out-of-vocabulary name errors in speech recognition create significant problems for downstream language processing, but the fact that they are rare poses challenges for automatic detection, particularly in an open-domain scenario. To address this problem, a multi-task recurrent neural network language model for sentence-level name detection is proposed for use in combination with out-of-vocabulary word detection. The sentence-level model is also effective for leveraging external text data. Experiments show a 26% improvement in name-error detection F-score over a system using n-gram lexical features.
BibTeX
@inproceedings{cheng2015open,
  author = {Hao Cheng and Hao Fang and Mari Ostendorf},
  title = {{Open-Domain Name Error Detection using a Multi-Task RNN}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2015},
  url = {https://openreview.net/forum?id=SyWMGGMO-H},
}
Figure