Efficient combined approach for named entity recognition in spoken language

Azeddine Zidouni, Sophie Rosset, Hervé GlotinInterspeech 2010

TL;DR

This paper presents a novel approach to Named Entity Recognition (NER) in spoken language by combining symbolic linguistic analysis with statistical machine learning (Conditional Random Fields - CRF). This hybrid method significantly improves NER accuracy on noisy Automatic Speech Recognition (ASR) outputs compared to purely statistical or symbolic methods, making it more robust for real-world spoken language applications.

What Problem Does This Paper Address?

Named Entity Recognition (NER) is crucial for information extraction, aiming to identify and categorize proper nouns (like people, organizations, locations) and numerical expressions in text. While NER on clean text has seen great progress, applying it to spoken language, especially when derived from Automatic Speech Recognition (ASR) systems, presents a significant challenge. ASR outputs are inherently noisy, containing errors from out-of-vocabulary words and acoustic/language model mismatches. These errors can cascade and severely degrade the performance of downstream NER systems. Existing approaches often struggle to maintain accuracy in the face of this noise, leading to a gap between performance on clean text and spoken language.

Methods

The authors propose a combined approach that leverages the strengths of both symbolic and statistical methods.

  • Symbolic Analyzer (LIMSI): This component, based on the Ritel system, uses a rule-based approach with a multi-level analyzer. It employs a tool called WMatch, which allows for matching regular expressions using words as the base unit, enabling the use of word classes and macros. This system is designed for spoken language dialog and question-answering, and its entity definitions were adapted to the ESTER 2 task.
  • Statistical Annotator (LSIS): This is a Conditional Random Fields (CRF) based approach. CRFs are powerful graphical models for sequence labeling tasks. They are discriminative and allow for the incorporation of a wide variety of features, including word attributes, neighboring words, and, crucially in this work, the output of the symbolic analyzer. The CRF model uses a context window and incorporates features like Part-of-Speech (POS) tags and semantic information from the Ritel system.
  • Fusion Protocol: The core innovation lies in fusing the outputs of the LIMSI symbolic analyzer as features into the LSIS CRF model. Instead of relying solely on traditional features like word capitalization or prefixes/suffixes, the CRF model is enriched with semantic information derived from the symbolic analysis. This allows the statistical model to benefit from linguistic insights that might be missed in noisy ASR data.

Datasets

The experiments were conducted using the ESTER 2 corpus. This corpus consists of 100 hours of manually transcribed French broadcast news from four radio channels. It includes a NER task with seven main categories (person, location, organization, time, amount, function, production) and over 30 sub-categories. The corpus was divided into training (84%), development (8%), and testing (8%) sets. Evaluation was performed on both manual transcriptions (testREF) and three different ASR outputs, with a focus on the first ASR output (testASR) which had a Word Error Rate (WER) of 12.11%.

Key Insights and Ideas

  • Synergy of Symbolic and Statistical: The paper strongly demonstrates that combining symbolic linguistic analysis with statistical models like CRFs is more effective for NER in noisy spoken language than either approach alone. The symbolic system provides valuable semantic context that helps the CRF model overcome ASR errors.
  • Robustness to ASR Noise: The proposed combined approach shows significantly improved robustness on ASR outputs compared to baseline systems. This is crucial for practical applications that rely on spoken input.
  • Feature Engineering Matters: Enriching CRF models with domain-specific semantic features derived from symbolic analysis can lead to substantial performance gains, especially for certain entity types.
  • Class-Specific Sensitivity: Some entity types (like FUNC and ORG) benefit more from semantic features than others, suggesting that the nature of the entity and the available semantic triggers play a role.
  • Manual vs. ASR Performance Gap: The results highlight that systems performing well on manual transcriptions are not necessarily the most robust on ASR output, underscoring the need for specialized approaches for spoken language NER.

Key Results

The combined CRFRitel system achieved the 3rd best Slot Error Rate (SER) on manual transcriptions (20.26%) among the evaluated systems, showing a significant improvement over the baseline LSIS system (34.98%) and the LIMSI system (30.88%). On ASR output, the CRFRitel system also demonstrated robustness, achieving a slight improvement over the best system. Specifically, the paper highlights:

  • Overall SER Improvement: The CRFRitel system achieved an overall SER of 20.26% on manual transcriptions, a substantial reduction from the LSIS system's 34.98%.
  • Specific Entity Gains: Significant SER reductions were observed for LOC (-24.80%), ORG (-21.04%), and FUNC (-17.06%) NE types compared to the CRF baseline.
  • Robustness on ASR: The CRFRitel approach showed a slight improvement (0.25%) over the best system on ASR output, indicating its resilience to speech recognition errors.

Limitations and Future Work

  • PROD Quality Degradation: The paper notes that the PROD (production) entity type's quality degraded with the CRFRitel system. This is attributed to a lack of sufficient examples in the corpus to fully optimize its parameters.
  • Ambiguous Classes: Confusion between ambiguous classes like ORG and LOC remains a challenge, suggesting that lexical and syntactic information, beyond just semantic context, are important for disambiguation.
  • Future Work: The authors suggest several directions:
  • Optimizing context length for different NE types.
  • Extending the approach to other semantic annotations of noisy inputs.
  • Introducing more speech-specific features, such as harmonic-to-noise ratios (HNR).

Summary updated

BibTeX
@inproceedings{zidouni2010efficient,
  author = {Azeddine Zidouni and Sophie Rosset and Hervé Glotin},
  title = {{Efficient combined approach for named entity recognition in spoken language}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure