TL;DR
This paper tackles the challenge of accurately extracting Named Entities (NEs) from large spoken archives, which are often plagued by noisy Automatic Speech Recognition (ASR) transcripts and temporal mismatches between training and test data. The authors propose a novel ASR-integrated Named Entity Recognition (NER) system that leverages word lattices and metadata to significantly improve NE extraction performance in these difficult conditions.
What Problem Does This Paper Address?
Extracting structured information, like Named Entities (people, locations, organizations), from spoken audio is a crucial task for many applications, such as information retrieval and content analysis. Traditional approaches often involve first transcribing the speech using an Automatic Speech Recognition (ASR) system and then applying text-based Named Entity Recognition (NER) methods to the transcript. However, this pipeline is highly sensitive to the quality of the ASR transcript. When the Word Error Rate (WER) of the transcript is high, NER performance degrades dramatically. This is particularly problematic for large spoken archives, which often contain audio recorded under diverse conditions, leading to significant temporal mismatches between the data used to train the ASR and NER models and the actual archive content. Proper names, a core component of NEs, are also highly time-dependent, exacerbating the impact of these mismatches. The paper aims to develop a more robust NER system that can handle these challenges effectively.
Methods
The paper introduces two NER systems:
- NERtext: A baseline system using the off-the-shelf NLP tool Lingpipe, trained on text input. This serves as a benchmark for comparison.
- NERasr: The core contribution of the paper. This system is specifically designed for ASR output and has two key features:
- Word Lattice Processing: Instead of relying on the single "best" ASR hypothesis (1-best output), NERasr processes word lattices. Word lattices represent multiple possible transcriptions, offering a richer search space that can mitigate errors in the ASR's top choice. This is achieved by composing Finite State Machine (FSM) transducers representing NE grammars with the ASR output FSM.
- ASR Lexicon Integration: The system leverages the fact that ASR systems operate with a closed vocabulary. NER models are trained using this specific ASR lexicon, allowing for over-training on known words and better control over generalization, which can be detrimental in noisy ASR output.
- Two-Level Approach: NERasr employs a two-level architecture: NE grammars (FSM transducers) for initial candidate extraction and a statistical HMM-based tagger for refining decisions.
The paper also explores metadata adaptation. They hypothesize that newsletters from the same time period as the spoken archive can be used to adapt both the ASR language model (reducing Out-Of-Vocabulary words) and the NER models (enriching wordlists and grammars).
Datasets
- ESTER Corpus: A French Rich Transcription program corpus of 100 hours of Broadcast News data. This corpus was manually transcribed and labeled with approximately 30 Named Entity categories.
- Matched Corpus: Training and development sets with a temporal overlap.
- Unmatched Corpus: Test set with a 6-month temporal gap from the training data, introducing a temporal mismatch.
- Le Monde Newsletters: A corpus of daily newsletters from the French newspaper Le Monde, collected over 6 months. This was used as metadata for adaptation.
Key Insights and Ideas
- Word lattices are crucial for robust NER on noisy ASR: Moving beyond the 1-best ASR output to word lattices significantly improves NER performance, especially in challenging conditions.
- Leveraging the ASR's closed vocabulary is a key advantage: Unlike text-based NER, which relies heavily on generalization for unknown words, ASR-based NER can benefit from over-training on the ASR's lexicon, leading to better control and reduced spurious detections.
- Temporal mismatch is a major hurdle for NEs: Proper names are highly time-dependent, making NER performance drop significantly when there's a temporal gap between training and test data.
- Metadata can offer targeted improvements: While general metadata adaptation might yield small gains, focusing on metadata directly relevant to the test data (e.g., newsletters from the exact same days) can lead to substantial improvements in recall for specific indexing tasks.
Key Results
- NERasr outperforms NERtext on ASR output: On the unmatched corpus, NERasr achieved a higher F-measure (73%) compared to NERtext (72%) on ASR transcripts, demonstrating its robustness.
- Significant recall improvement with metadata adaptation: When using newsletters from the same period as the test data for adaptation, the recall for specific indexing terms increased from 75.7% to 83.9% (an absolute increase of 8.2 percentage points), while precision also saw a slight increase.
- Oracle recall shows potential: The Oracle recall measure (an upper bound on performance) for NERasr on the unmatched corpus was significantly higher (76.9%) than for NERtext (61.9%), indicating the potential for further improvement with better decision strategies.
- Metadata adaptation has limited overall impact: General adaptation using 6 months of newsletters showed only a 0.3% absolute WER reduction and a similar decrease in SER, suggesting that broad adaptation might not be sufficient for significant gains.
Limitations and Future Work
- Overall metadata adaptation impact is limited: The paper notes that the general adaptation using a 6-month newsletter corpus did not yield significant improvements in ASR or NER performance.
- Future work on decision strategies: The authors are exploring more elaborate decision strategies for selecting the best NE hypothesis from the n-best lists generated by NERasr, aiming to fully leverage the information in word lattices.
- Further investigation into metadata's role: While specific metadata adaptation showed promise for indexing, further research is needed to fully understand how to best exploit metadata for general NER improvement.
Summary updated