Abstract
Named Entity Recognition (NER) from speech is a critical component of Spoken Language Understanding, yet it faces significant challenges such as cascading errors from Automatic Speech Recognition (ASR) and phonetic ambiguities. This paper introduces AISHELL-NER, the first large-scale open-source dataset for Chinese speech NER, comprising over 170 hours of annotated Mandarin speech. We evaluate several architectures, including traditional pipelines, end-to-end (E2E) entity-aware models, and a hybrid approach. Our results show that while E2E models mitigate error propagation, a hybrid system combining a Conformer-based entity-aware ASR with a pretrained BERT tagger achieves the highest performance (74.90 F1), highlighting the importance of large-scale text pretraining in overcoming the "homophone problem" inherent in Chinese speech.
Introduction
Named Entity Recognition (NER) involves identifying and categorizing key information—such as people (PER), locations (LOC), and organizations (ORG)—within text. In the context of speech, this task is traditionally handled via a pipeline: an ASR system converts audio to text, which is then processed by an NER tagger. However, this approach suffers from cascading errors, where ASR inaccuracies prevent the NER system from correctly identifying entities.
Chinese speech presents a unique challenge: homophones. Many distinct characters share the same pronunciation (e.g., "Jin" could refer to a city name or a common noun). Without sufficient context, standard ASR systems may select the wrong character even if the phonetic transcription is accurate, leading to NER failure. This paper addresses these challenges by introducing a new dataset and exploring robust modeling strategies.
The AISHELL-NER Dataset
To facilitate research in this area, the authors introduce AISHELL-NER, built upon the AISHELL-1 Mandarin speech corpus. The dataset includes:
- Scale: Over 170 hours of Mandarin speech.
- Volume: Approximately 120,000 training sentences across domains including Finance, Science, Sports, Entertainment, and News.
- Annotations: Manual labels for PER, LOC, and ORG entities, provided in both transcript-with-tags and standard BIO formats.
Methodology
The researchers explored three primary architectural paradigms to bridge the gap between speech and entity recognition:
- The Pipeline: A standard ASR (Transformer or Conformer-based) followed by a BERT-based NER tagger.
- End-to-End (E2E) / Entity-Aware ASR (EA-ASR): The ASR system is trained to insert special entity tokens directly into the transcript (e.g.,
[Guo Jing]PER). This allows the model to learn direct alignments between audio signals and entities. - The Hybrid Approach: An Entity-Aware ASR generates a "rich" transcript containing entity hints, which is then refined by a pretrained BERT NER tagger.
The ASR components utilized the Conformer architecture, which integrates the global modeling capabilities of Transformers with the local feature extraction of Convolutional Neural Networks (CNNs).
Experimental Results
Experimental evaluations revealed several key insights regarding model performance and the nature of spoken NER:
- Conformer Superiority: Conformer-based ASR significantly outperformed standard Transformers, achieving a Character Error Rate (CER) of 4.75% on the test set compared to 9.16% for the Transformer.
- The Impact of Pretraining: In the pipeline setup, utilizing a pretrained BERT (BERT-PT) tagger boosted the F1 score from approximately 69% to 74.04%, demonstrating the value of linguistic knowledge gained from massive text corpora.
- Optimal Strategy: The hybrid approach—combining a Conformer EA-ASR with a pretrained BERT tagger—achieved the state-of-the-art F1 score of 74.90. This suggests that the entity "hints" from the EA-ASR help the BERT tagger resolve ambiguities more effectively.
Key Insights
- The Pretraining Paradox: While E2E models are theoretically more elegant by avoiding cascading errors, they currently struggle to match the performance of pipeline/hybrid models that leverage massive text-only pretraining (like BERT).
- The Homophone Bottleneck: The largest performance drops occurred in the PER category. Chinese names often use rare characters that sound like common words; if the ASR "misspells" the name despite correct phonetics, the NER system typically fails.
- ASR vs. Entity Awareness: Interestingly, training the ASR to be entity-aware slightly increased the CER (from 4.75% to 4.79% for Conformer). This indicates a minor trade-off where the model's focus on identifying entities can slightly distract from simple transcription accuracy.
Conclusion and Future Work
While the proposed hybrid model shows significant promise, a substantial gap remains between NER performance on ground-truth text (93.26 F1) and speech (74.90 F1). The primary obstacle remains the homophone problem in Chinese nomenclature. Future work will focus on developing pretrainable E2E models that can ingest both audio and text data simultaneously and exploring phonetically-aware evaluation metrics that do not penalize homophone substitutions as harshly as unrelated transcription errors.
Summary updated