Noise Robust Named Entity Understanding for Voice Assistants

Deepak Muralidharan, Joel Ruben Antony Moniz, Sida Gao, Xiao Yang, Justine T. Kao, Stephen Pulman, Atish Kothari, Ray Shen, Yinying Pan, Vivek Kaul, Mubarak Seyed Ibrahim, Gang Xiang, Nan Dun, Yidan Zhou, Andy O, Yuan Zhang, Pooja Chitkara, Xuan Wang, Alkesh Patel, Kushal Tayal, Roger Zheng, Peter Grasch, Jason D. Williams, Lin Li

Abstract

Researchers at Apple have developed a "Named Entity Understanding" (NEU) framework that jointly optimizes Named Entity Recognition (NER) and Entity Linking (EL) to handle the messy reality of voice assistant queries. By using a joint reranking module and character-level neural networks, the system significantly improves accuracy on noisy spoken data, outperforming traditional pipelined approaches. (Source: Muralidharan et al., NAACL 2021 Industry Papers).

Introduction

Voice assistants like Siri or Alexa face a unique "noise" problem. Unlike written text, spoken queries are often extremely short, averaging just five tokens, which provides very little context for disambiguation. Furthermore, the input is prone to three types of errors: Automatic Speech Recognition (ASR) errors (e.g., "Bohemian raspberry" instead of "Rhapsody"), user errors (e.g., "Cristiano Nando" instead of "Ronaldo"), and the use of non-canonical forms or aliases.

In a traditional pipeline, NER identifies a "thing" (e.g., a Person), and EL then tries to link that "thing" to a specific entry in a Knowledge Base (KB). If the NER step fails due to noise, the EL step has no chance of succeeding. This paper proposes a way to let these two tasks interact to fix errors through a joint reranking architecture.

Methodology

The authors propose a Named Entity Understanding (NEU) architecture that moves away from a rigid pipeline toward a joint reranking approach.

NER Module and Candidate Retrieval

The system uses a standard biLSTM-CRF model with character and word-level (GloVe) features. Instead of passing a single best guess, it employs beam search to generate the top five NER hypotheses. For each hypothesis, a retrieval engine queries the Knowledge Base. To handle noise, the query passes through a seq-to-seq correction model and uses an inverted index that includes common aliases and n-grams.

Joint Linking and Reranking

This is the core innovation: a neural ranker evaluates the NER hypotheses and EL candidates simultaneously.

  • Character-level CNNs: To be robust against misspellings or ASR errors, CNNs create embeddings of the mentions and the KB entities. Because "raspberry" and "rhapsody" look similar at a character level, the model can learn to associate them even if their word embeddings are distinct.
  • Relational and Popularity Features: The model looks for relationships in the KB. If a query mentions two entities belonging to the same sports league or music album, this "relational signal" boosts confidence. It also incorporates "prior usage probability" to account for entity popularity.

Key Results

The NEU framework showed consistent improvements across high-entity domains including Music, Sports, and Movie & TV.

Overall Performance

NER accuracy improved by up to 3.13% in F1 score through reranking, while EL accuracy saw gains of up to 3.6%.

Table showing NER F1 (top-1 and top-5), reranked NER F1, and EL F1 scores across domains.

Robustness to Noise

The system's strength is most apparent in noisy conditions. On the "noisy" subset of the music domain, using the MLP-based similarity feature improved EL performance by a staggering 10.03%.

Table illustrating the performance boost (+10.03%) on noisy music and movie&TV subsets using MLP-based similarity.

Downstream Integration

Adding NEU features improved performance in downstream tasks. Domain Classification accuracy rose (e.g., from 88.95 to 90.04), and Semantic Parsing accuracy improved across all tested domains, demonstrating the utility of the framework in a production-style environment.

Conclusion

The NEU framework demonstrates that Entity Linking should inform NER rather than just follow it. By using character-level features and KB relational data, the system achieves significant robustness to the ASR and user errors common in voice assistant interactions. Future work will focus on addressing the "cold start" problem for entities missing from the KB and further optimizing inference latency.

Summary updated

BibTeX
@inproceedings{muralidharan2021noise,
  author = {Deepak Muralidharan and Joel Ruben Antony Moniz and Sida Gao and Xiao Yang and Justine T. Kao and Stephen Pulman and Atish Kothari and Ray Shen and Yinying Pan and Vivek Kaul and Mubarak Seyed Ibrahim and Gang Xiang and Nan Dun and Yidan Zhou and Andy O and Yuan Zhang and Pooja Chitkara and Xuan Wang and Alkesh Patel and Kushal Tayal and Roger Zheng and Peter Grasch and Jason D. Williams and Lin Li},
  title = {{Noise Robust Named Entity Understanding for Voice Assistants}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=wje_BwRtRyb},
}
Figure