AISHELL-NER: Named Entity Recognition from Chinese Speech

Boli Chen, Guangwei Xu, Xiaobin Wang, Pengjun Xie, Meishan Zhang, Fei Huang

Abstract

Named Entity Recognition (NER) from speech is a critical component of Spoken Language Understanding, yet it faces significant challenges such as cascading errors from Automatic Speech Recognition (ASR) and phonetic ambiguities. This paper introduces AISHELL-NER, the first large-scale open-source dataset for Chinese speech NER, comprising over 170 hours of annotated Mandarin speech. We evaluate several architectures, including traditional pipelines, end-to-end (E2E) entity-aware models, and a hybrid approach. Our results show that while E2E models mitigate error propagation, a hybrid system combining a Conformer-based entity-aware ASR with a pretrained BERT tagger achieves the highest performance (74.90 F1), highlighting the importance of large-scale text pretraining in overcoming the "homophone problem" inherent in Chinese speech.

Introduction

Named Entity Recognition (NER) involves identifying and categorizing key information—such as people (PER), locations (LOC), and organizations (ORG)—within text. In the context of speech, this task is traditionally handled via a pipeline: an ASR system converts audio to text, which is then processed by an NER tagger. However, this approach suffers from cascading errors, where ASR inaccuracies prevent the NER system from correctly identifying entities.

Chinese speech presents a unique challenge: homophones. Many distinct characters share the same pronunciation (e.g., "Jin" could refer to a city name or a common noun). Without sufficient context, standard ASR systems may select the wrong character even if the phonetic transcription is accurate, leading to NER failure. This paper addresses these challenges by introducing a new dataset and exploring robust modeling strategies.

The AISHELL-NER Dataset

To facilitate research in this area, the authors introduce AISHELL-NER, built upon the AISHELL-1 Mandarin speech corpus. The dataset includes:

  • Scale: Over 170 hours of Mandarin speech.
  • Volume: Approximately 120,000 training sentences across domains including Finance, Science, Sports, Entertainment, and News.
  • Annotations: Manual labels for PER, LOC, and ORG entities, provided in both transcript-with-tags and standard BIO formats.

Methodology

The researchers explored three primary architectural paradigms to bridge the gap between speech and entity recognition:

  1. The Pipeline: A standard ASR (Transformer or Conformer-based) followed by a BERT-based NER tagger.
  2. End-to-End (E2E) / Entity-Aware ASR (EA-ASR): The ASR system is trained to insert special entity tokens directly into the transcript (e.g., [Guo Jing]PER). This allows the model to learn direct alignments between audio signals and entities.
  3. The Hybrid Approach: An Entity-Aware ASR generates a "rich" transcript containing entity hints, which is then refined by a pretrained BERT NER tagger.
Comparison of Pipeline and End-to-End (E2E) NER architectures.

The ASR components utilized the Conformer architecture, which integrates the global modeling capabilities of Transformers with the local feature extraction of Convolutional Neural Networks (CNNs).

Experimental Results

Experimental evaluations revealed several key insights regarding model performance and the nature of spoken NER:

  • Conformer Superiority: Conformer-based ASR significantly outperformed standard Transformers, achieving a Character Error Rate (CER) of 4.75% on the test set compared to 9.16% for the Transformer.
  • The Impact of Pretraining: In the pipeline setup, utilizing a pretrained BERT (BERT-PT) tagger boosted the F1 score from approximately 69% to 74.04%, demonstrating the value of linguistic knowledge gained from massive text corpora.
  • Optimal Strategy: The hybrid approach—combining a Conformer EA-ASR with a pretrained BERT tagger—achieved the state-of-the-art F1 score of 74.90. This suggests that the entity "hints" from the EA-ASR help the BERT tagger resolve ambiguities more effectively.
Performance comparison of various models on the AISHELL-NER test set.

Key Insights

  • The Pretraining Paradox: While E2E models are theoretically more elegant by avoiding cascading errors, they currently struggle to match the performance of pipeline/hybrid models that leverage massive text-only pretraining (like BERT).
  • The Homophone Bottleneck: The largest performance drops occurred in the PER category. Chinese names often use rare characters that sound like common words; if the ASR "misspells" the name despite correct phonetics, the NER system typically fails.
  • ASR vs. Entity Awareness: Interestingly, training the ASR to be entity-aware slightly increased the CER (from 4.75% to 4.79% for Conformer). This indicates a minor trade-off where the model's focus on identifying entities can slightly distract from simple transcription accuracy.

Conclusion and Future Work

While the proposed hybrid model shows significant promise, a substantial gap remains between NER performance on ground-truth text (93.26 F1) and speech (74.90 F1). The primary obstacle remains the homophone problem in Chinese nomenclature. Future work will focus on developing pretrainable E2E models that can ingest both audio and text data simultaneously and exploring phonetically-aware evaluation metrics that do not penalize homophone substitutions as harshly as unrelated transcription errors.

Summary updated

Abstract

Named Entity Recognition (NER) from speech is among Spoken Language Understanding (SLU) tasks, aiming to extract semantic information from the speech signal. NER from speech is usually made through a two-step pipeline that consists of (1) processing the audio using an Automatic Speech Recognition (ASR) system and (2) applying an NER tagger to the ASR outputs. Recent works have shown the capability of the End-to-End (E2E) approach for NER from English and French speech, which is essentially entity-aware ASR. However, due to the many homophones and polyphones that exist in Chinese, NER from Chinese speech is effectively a more challenging task. In this paper, we introduce a new dataset AISEHLL-NER for NER from Chinese speech. Extensive experiments are conducted to explore the performance of several state-of-the-art methods. The results demonstrate that the performance could be improved by combining entity-aware ASR and pretrained NER tagger, which can be easily applied to the modern SLU pipeline. The dataset is publicly available at github.com/Alibaba-NLP/AISHELL-NER.
BibTeX
@inproceedings{chen2022aishell,
  author = {Boli Chen and Guangwei Xu and Xiaobin Wang and Pengjun Xie and Meishan Zhang and Fei Huang},
  title = {{AISHELL-NER: Named Entity Recognition from Chinese Speech}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=qi5bFwm-KM},
}
Figure