Detecting OOV named-entities in conversational speech

Rohit Kumar, Rohit Prasad, Sankaranarayanan Ananthakrishnan, Aravind Namandi Vembu, Dave Stallard, Stavros Tsakalidis, Prem NatarajanInterspeech 2012

TL;DR

This paper tackles the challenge of accurately identifying named entities (like people, places, and organizations) in conversational speech, especially when these entities are "out-of-vocabulary" (OOV) for the Automatic Speech Recognition (ASR) system. They propose using ASR's internal confidence scores (word posteriors) to improve named entity detection, finding that this can significantly boost OOV detection but might slightly degrade performance for in-vocabulary entities.

What Problem Does This Paper Address?

Speech-to-speech translation systems are powerful tools for breaking down language barriers. However, their performance can be severely hampered by errors from their underlying components, particularly Automatic Speech Recognition (ASR). A common ASR error is failing to recognize words that aren't in its vocabulary, known as "out-of-vocabulary" (OOV) words. Named Entities (NEs) – such as names of people, places, and organizations – are a critical type of word that frequently falls into the OOV category. While misrecognizing a regular OOV word might only slightly increase the Word Error Rate (WER), misrecognizing an OOV NE can completely derail a conversation, as it represents a failure in understanding crucial information. This paper aims to improve the detection of these OOV NEs in conversational speech, even when the ASR system itself cannot correctly transcribe them.

Methods

The core of the proposed approach is a Maximum Entropy (MaxEnt) classifier, a standard machine learning technique for classification tasks. The novelty lies in how they incorporate information from the ASR system into this classifier.

  • Baseline: They start with a MaxEnt model that uses standard features like n-gram context, part-of-speech (POS) tags (both supervised for English and unsupervised for Iraqi Arabic), and the word itself.
  • ASR Word Posteriors: The key innovation is leveraging the "word posteriors" generated by the ASR system. These posteriors represent the ASR's confidence in its transcription of each word. The intuition is that OOV NEs, when misrecognized, are likely to be associated with low confidence scores from the ASR.
  • Early Fusion: They integrate ASR word posteriors by using them as weights for the features fed into the MaxEnt classifier. Features derived from words with low posteriors are given less weight, effectively downplaying uncertain information. This weighting is applied post-hoc, meaning it's used during the detection phase without retraining the ASR or the initial MaxEnt model.
  • Late Fusion: This approach combines the decisions of multiple detectors. It uses the early fusion detector and a separate detector based on word posterior minima. If the early fusion detector is highly confident, its decision is used. If its confidence is moderate, it backs off to a stricter criterion, requiring agreement between the two detectors.

Datasets

The experiments were conducted on the DARPA TRANSTAC speech-to-speech collection, which includes English and Iraqi Arabic data.

  • English:
    • Training: Approximately 472,000 utterances from various sources, including web text. 137,000 of these contained NEs. NEs were tagged automatically using a lexicon.
  • Test: A held-out set of 2,374 spoken English utterances from TRANSTAC, with 118 manually tagged NE tokens.
  • Iraqi Arabic:
    • Training: 186,000 utterances from an English-to-Iraqi translation dataset, with manually tagged NEs. 9,000 utterances contained NEs (1185 unique tokens).
  • Test: 206 spoken Iraqi Arabic utterances from the TRANSTAC Name Evaluation, with 223 unique NE tokens (occurring 555 times). Crucially, 164 of these unique tokens were OOV for the NER system.

For evaluating the impact of ASR errors, they also created an "OASR" (Out-of-Vocabulary ASR) condition by removing NE tokens from the ASR's language model and lexicon, simulating a worst-case scenario where NEs are completely unknown to the ASR.

Key Insights and Ideas

  • OOV NEs are a critical failure point: The paper strongly emphasizes that while OOV words might slightly increase WER, OOV NEs can cause complete communication breakdowns in speech translation systems.
  • ASR confidence is a valuable signal for NE detection: The core insight is that the ASR's internal confidence scores (word posteriors) can be a powerful indicator of potential NEs, especially OOV ones. Low confidence suggests uncertainty, which is often associated with misrecognized NEs.
  • Feature weighting can improve OOV detection: Using word posteriors to weight features in an early fusion approach significantly boosts the detection of OOV NEs. This suggests that down-weighting features derived from uncertain ASR outputs is beneficial.
  • The trade-off between OOV and IV NE detection: A crucial finding is that techniques designed to improve OOV NE detection can sometimes hurt the detection of in-vocabulary (IV) NEs. This is because IV NEs are typically recognized with high confidence, and the proposed methods might inadvertently penalize them. The choice of features should therefore consider the expected OOV rate.

Key Results

  • Improved OOV NE Detection: For the English OASR (100% OOV) condition, the proposed techniques (Early and Late Fusion) improved NE detection rates by 3-11% over baseline methods, particularly in the operating range of 7-12% Word False Alarm (wFA).
  • Performance Degradation on IV NEs: When applied to the ASR set (where most NEs are in-vocabulary), these OOV-focused techniques led to a decrease in detection rates compared to baseline methods.
  • Language Differences: The "current word" feature was found to be more beneficial for Iraqi Arabic, where a higher proportion of NEs are in-vocabulary, compared to English.
  • Table 2 & 3 Summary:
    • English: MaxEnt with context, POS, and word features achieved a Word Detection Rate (wDet) of 77.7% (Ref) and 36.1% (OASR). Early Fusion reached 57.0% (OASR), and Late Fusion reached 54.6% (OASR).
  • Iraqi Arabic: MaxEnt with context, POS, and word features achieved a wDet of 95.9% (Ref) and 53.7% (ASR). Early Fusion reached 52.6% (ASR), and Late Fusion reached 53.1% (ASR). (Note: OASR results for Iraqi Arabic were not explicitly detailed in the tables but implied by the text).

Limitations and Future Work

  • Deterioration of IV NE detection: The authors acknowledge that their proposed techniques, while improving OOV NE detection, can negatively impact the detection of in-vocabulary NEs.
  • Future Integration Strategies: The authors plan to investigate techniques for more deeply integrating ASR cues into NER models, moving beyond post-hoc weighting, to improve OOV detection without sacrificing IV detection performance, regardless of the OOV rate.
  • Interaction Strategies: They are also developing interaction strategies for speech-to-speech translation systems that leverage named entity detectors to proactively identify and correct potential misrecognition errors, thereby preventing communication failures. These strategies would consider triggered detections and conversational context.

Summary updated

BibTeX
@inproceedings{kumar2012detecting,
  author = {Rohit Kumar and Rohit Prasad and Sankaranarayanan Ananthakrishnan and Aravind Namandi Vembu and Dave Stallard and Stavros Tsakalidis and Prem Natarajan},
  title = {{Detecting OOV named-entities in conversational speech}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure