Exploring Named Entity Recognition As an Auxiliary Task for Slot Filling in Conversational Language Understanding

Samuel Louvan, Bernardo MagniniEMNLP 2018

TL;DR

This paper explores using Named Entity Recognition (NER) as an auxiliary task to improve slot filling performance in conversational AI. They found that jointly training NER and slot filling, especially when NER is supervised at lower layers of the model, consistently boosts slot filling accuracy. This approach offers a way to leverage readily available NER data to enhance performance on the more domain-specific slot filling task.

What Problem Does This Paper Address?

Conversational AI systems rely on Natural Language Understanding (NLU) to extract meaning from user utterances. A key component of NLU is slot filling, which involves identifying and extracting specific pieces of information (slots) that correspond to a user's intent. Traditionally, slot filling models are trained on domain-specific datasets, which can be costly and time-consuming to create, especially for low-resource domains. This paper addresses the challenge of improving slot filling performance, particularly in scenarios where labeled data is scarce, by investigating the effectiveness of multi-task learning (MTL) with Named Entity Recognition (NER) as an auxiliary task. The hypothesis is that NER, which identifies general entity types (like person, location, organization), can provide valuable, transferable knowledge to the more granular slot filling task.

Methods

The authors propose a multi-task learning (MTL) framework where a single neural network is trained to perform both slot filling and NER simultaneously. The core of their model is a bidirectional LSTM (biLSTM) followed by a Conditional Random Field (CRF) layer, a standard architecture for sequence labeling tasks like NER and slot filling.

Key methodological aspects include:

  • Shared Lower Layers: The biLSTM encoder, which processes word and character embeddings, is shared between the slot filling and NER tasks. This allows the model to learn general feature representations that benefit both tasks.
  • Task-Specific Output Layers: Separate CRF layers are used for slot filling and NER to predict task-specific tags.
  • Supervision Levels: They investigate two MTL setups:
  • Same Supervision Level: Both tasks are supervised at the same layer (typically the output layer).
  • Different Supervision Level: NER is supervised at a lower layer of the biLSTM encoder, while slot filling is supervised at a higher layer. This is based on the intuition that more general features (learned at lower layers) are better for auxiliary tasks like NER, which can then inform the more specific slot filling task.
  • Data Selection: They explore techniques for selecting auxiliary data:
  • Domain Similarity: Using Jensen-Shannon divergence to find the most similar domain for the auxiliary task.
  • Label Embedding Mapping: Using Canonical Correlation Analysis (CCA) to map label embeddings and select relevant auxiliary labels.

Datasets

The paper uses the following datasets:

  • Slot Filling Datasets:
    • ATIS: Airline Travel Information System (queries to a flight database).
  • MIT Restaurant (MIT-R): Utterances related to restaurant search.
  • MIT Movie (MIT-M): Queries related to movie information.
  • NER Datasets:
    • CONLL 2003: A widely used dataset for NER.
  • OntoNotes 5.0: Another comprehensive NER dataset.

The authors use standard splits for these datasets and mention that they use the Newswire section of OntoNotes.

Key Insights and Ideas

  • NER as a Powerful Auxiliary Task: The core finding is that NER significantly improves slot filling performance, especially when integrated via MTL. This suggests that the general entity recognition capabilities learned from NER are highly transferable to the more specific slot filling task.
  • Lower-Layer Supervision is Key: Supervising NER at the lower layers of the biLSTM encoder proved more effective than same-level supervision. This aligns with the idea that general features learned early in the network are more beneficial for auxiliary tasks and can then propagate to improve the main task.
  • Domain Similarity Matters for Auxiliary Tasks: When using another slot filling domain as an auxiliary task, selecting the most similar domain was found to be less effective than using all available domains. However, when NER was combined with the most similar domain, it outperformed using all domains. This highlights the distinct value of NER's general knowledge.
  • MTL's Potential in Low-Resource Scenarios: While not universally effective across all datasets in low-resource settings, MTL showed promise, particularly for the MIT-R dataset, suggesting it can help mitigate data scarcity.

Key Results

  • Consistent Improvement: Using NER as an auxiliary task consistently improved slot filling performance across all tested datasets.
  • Lower-Layer NER Advantage: The "different supervision level" setup (NER at lower layers) generally yielded better results than "same supervision level."
  • Specific Slot Improvements: Analysis on development sets showed MTL models were better at recognizing location-related slots, while single-task models sometimes performed better on time-related slots.
  • Low-Resource Performance: In low-resource scenarios (200, 400, 800 training sentences), MTL consistently outperformed the single-task baseline for MIT-R. For ATIS and MIT-M, the single-task model often performed better, though MTL still showed competitive results.

Limitations and Future Work

  • Small Performance Gains: While NER improved performance, the difference was sometimes "rather small," particularly for ATIS and MIT Movie datasets. The authors suggest further work is needed to explore better mechanisms for injecting NER information.
  • Data Selection Nuances: The effectiveness of domain similarity and label embedding mapping for data selection requires further investigation. The paper notes that filtering auxiliary labels using label embedding mapping actually dropped performance, suggesting the quality of mapping and the prevalence of the "O" label were issues.
  • Comparison with Pipeline Systems: The authors suggest comparing their MTL approach with pipeline systems that use NER predictions as features for slot filling.
  • Exploring Other Auxiliary Tasks: Investigating other general NLP tasks as auxiliary tasks for slot filling could be a fruitful direction.

Summary updated

BibTeX
@inproceedings{louvan2018exploring,
  author = {Samuel Louvan and Bernardo Magnini},
  title = {{Exploring Named Entity Recognition As an Auxiliary Task for Slot Filling in Conversational Language Understanding}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure