Abstract
Spoken language understanding (SLU) tasks involve mapping from speech audio signals to semantic labels. Given the complexity of such tasks, good performance might be expected to require large labeled datasets, which are difficult to collect for each new task and domain. However, recent advances in self-supervised speech representations have made it feasible to consider learning SLU models with limited labeled data. In this work we focus on low-resource spoken named entity recognition (NER) and address the question: Beyond self-supervised pre-training, how can we use external speech and/or text data that are not annotated for the task? We draw on a variety of approaches, including self-training, knowledge distillation, and transfer learning, and consider their applicability to both end-to-end models and pipeline (speech recognition followed by text NER model) approaches. We find that several of these approaches improve performance in resource-constrained settings beyond the benefits from pre-trained representations alone. Compared to prior work, we find improved F1 scores of up to 16%. While the best baseline model is a pipeline approach, the best performance when using external data is ultimately achieved by an end-to-end model. We provide detailed comparisons and analyses, showing for example that end-to-end models are able to focus on the more NER-specific words.
Introduction
Named Entity Recognition (NER)—identifying people, places, and organizations—is a cornerstone of natural language processing. However, performing this task directly from audio (Spoken NER) is significantly more challenging than from text. Speech signals are noisy, sequences are longer, and labeled datasets are notoriously difficult to collect. Currently, a 10-20% "performance gap" exists between NER performed on perfect text transcripts versus NER performed on speech.
Most research assumes the availability of hundreds of hours of labeled data. In real-world scenarios, however, researchers often only have a few hours of gold-standard annotations. This paper addresses the "low-resource" challenge: How can massive amounts of unlabeled speech and text be leveraged to improve spoken NER when only 15 hours of labeled data are available?
Architectural Approaches
The researchers compared two primary architectures for spoken NER:
- Pipeline: An Automatic Speech Recognition (ASR) module converts speech to text, which is then processed by a Text NER tagger.
- End-to-End (E2E): A single model that maps speech signals directly to NER-tagged text.
To boost these models using external (unlabeled) data, they employed three main strategies:
- Self-training: A baseline model labels unlabeled data; the model is then retrained on these "pseudo-labels."
- Knowledge Distillation: A strong "teacher" model (typically a pipeline system) generates labels for unlabeled speech to train a "student" E2E model, helping it learn from superior semantic understanding.
- Transfer Learning: Pre-training the E2E model on a standard ASR task before fine-tuning it for the specific NER task.
These models were built on state-of-the-art self-supervised representations: wav2vec 2.0 for speech and DeBERTa for text.
Experimental Setup
The study utilized the VoxPopuli dataset, specifically the SLUE Benchmark subset, providing a constrained 15-hour "fine-tune" set to simulate a low-resource environment. External data included 100-hour and 500-hour unlabeled subsets of VoxPopuli. Additionally, OntoNotes 5.0 was used to test if external, out-of-domain text NER data could improve the in-domain model.
Key Insights and Results
The most significant finding was the "Performance Flip": In the baseline setting without external data, the pipeline approach wins because the text NER component is pre-trained on massive amounts of text. However, once external data is introduced, the E2E model's performance scales much better, eventually surpassing the pipeline.
- E2E Superiority with External Data: The best E2E model (using 100h of external data via distillation) achieved an F1 score of 79.2, surpassing the best pipeline score of 75.6.
- Relative Gains: External data led to a 16% relative improvement for E2E models, compared to only 6% for pipeline models.
- Error Analysis: Pipeline models were found to be 1.5x more likely to hallucinate entities (false detections) than E2E models. This is often due to ASR phonetic errors that mislead the text NER module (e.g., transcribing "leverage" as "lever age").
- Domain Sensitivity: Interestingly, out-of-domain text NER data (OntoNotes) did not help, suggesting that acoustic and linguistic nuances of the specific domain (e.g., parliamentary speech) are more critical than general-purpose NER knowledge.
Conclusion
While the gap between spoken NER and NER on perfect transcripts (86.0 F1) remains, this work demonstrates that E2E models are "semantic sponges" that benefit significantly from knowledge distillation and external data. Future work will focus on improving pseudo-labeling quality to better utilize larger datasets and exploring why out-of-domain text data failed to provide benefits.
Reference: Pasad, A., Wu, F., Shon, S., Livescu, K., & Han, K. J. (2022). On the Use of External Data for Spoken Named Entity Recognition. arXiv:2112.07648v2 [cs.CL].
Summary updated