On the Use of External Data for Spoken Named Entity Recognition

Ankita Pasad, Felix Wu, Suwon Shon, Karen Livescu, Kyu J. Han

Abstract

Ankita Pasad, Felix Wu, Suwon Shon, Karen Livescu, Kyu Han. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.

2022.

Introduction

Named Entity Recognition (NER) is a fundamental task in Natural Language Processing, identifying key entities such as people, locations, and organizations. While text-based NER has reached high levels of maturity, spoken NER—extracting entities directly from audio—continues to lag significantly. This "speech gap," often resulting in performance drops of 10–20%, stems from the inherent complexities of spoken language, including disfluencies, the absence of punctuation or capitalization, and errors introduced by Automatic Speech Recognition (ASR) systems.

A primary obstacle in spoken NER is data scarcity. While text NER benefits from massive annotated corpora, labeled spoken NER datasets are often limited to just a few dozen hours. This research explores how to bridge this gap by leveraging external, unannotated speech and text through self-training and knowledge distillation, specifically examining how these resources benefit different model architectures.

Methods

The researchers evaluated two distinct architectural approaches for spoken NER:

  1. Pipeline Approach: A two-stage process where a speech recognizer (ASR) first converts audio into text, which is then processed by a standard, high-performance text NER model (such as DeBERTa).
  2. End-to-End (E2E) Approach: A unified model that maps raw audio signals directly to NER-tagged text outputs.
Overview of Text NER, Pipeline, and E2E NER architectures.

To improve performance in low-resource settings (using the 15-hour SLUE benchmark), the study employed several semi-supervised strategies:

  • Self-training: Using a baseline model to generate pseudo-labels for unannotated data, then retraining on those predictions.
  • Knowledge Distillation: Using a strong Pipeline "teacher" model to generate labels for unlabeled speech, which then trains an E2E "student" model.
  • Transfer Learning: Pre-training on related tasks like ASR before fine-tuning on the specific NER task.

The study utilized the VoxPopuli corpus for unlabeled speech and OntoNotes 5.0 for external text data to test the limits of cross-domain transfer.

Results and Key Insights

The experiments revealed that the choice of architecture significantly impacts how a model absorbs external data. A key finding is the "E2E Switch": while Pipeline models often perform better in baseline scenarios due to their reliance on powerful pre-trained text models, E2E models improve much more rapidly when external unlabeled audio is introduced.

F1 score comparison across different external data types, showing E2E models overtaking Pipeline models with external speech data.

Key results include:

  • Significant Performance Gains: The researchers achieved relative F1 score improvements of up to 16% for E2E models. On the development set, the E2E model's F1 score rose from a baseline of 68.1 to 82.2 when utilizing transcribed external speech.
  • Precision Advantages: Pipeline models were found to suffer more from "false detections" or hallucinations. In contrast, E2E models proved to be more conservative and precise in their entity identification.
  • Speech vs. Text Utility: Unlabeled speech (audio) was found to be far more valuable than unlabeled text. The acoustic information in speech helps the model better navigate the nuances and disfluencies of spoken language that text-only models may struggle with.
  • The Power of Distillation: The most effective method for boosting E2E performance was distillation from a Pipeline teacher, allowing the E2E model to inherit semantic knowledge while retaining its acoustic robustness.

Discussion and Conclusion

Despite these gains, the research identified several ongoing challenges. Increasing external data from 100 to 500 hours resulted in diminishing returns, suggesting that the quality of pseudo-labels may eventually become a bottleneck. Furthermore, the difficulty of cross-domain transfer (e.g., using OntoNotes for VoxPopuli speech) highlights that models still require data that closely matches the target task's domain.

In conclusion, this work demonstrates that by effectively leveraging unlabeled audio, E2E spoken NER models can not only close the gap with traditional pipeline systems but eventually surpass them, offering a more robust path forward for spoken language understanding in low-resource environments.

Summary updated

BibTeX
@inproceedings{pasad2022use,
  author = {Ankita Pasad and Felix Wu and Suwon Shon and Karen Livescu and Kyu J. Han},
  title = {{On the Use of External Data for Spoken Named Entity Recognition}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=BfXC4xw2te},
}
Figure