Learning Out-of-Vocabulary Words in Intelligent Personal Agents

Avik Ray, Yilin Shen, Hongxia JinIJCAI 2018 · OpenReview

Abstract

Semantic parsers are critical components of intelligent personal agents, translating natural language into actionable logical forms. However, these parsers often struggle with out-of-vocabulary (OOV) words and the "catastrophic forgetting" that occurs during retraining. This paper introduces novel neural architectures, including an Argument Transfer model and a hybrid paraphrase generator, designed to enhance robustness to OOV terms while maintaining performance on previously learned instructions. Experimental results on benchmark datasets demonstrate significant improvements in OOV handling and stability during continual learning.

Introduction

Intelligent agents like Siri or Alexa rely on semantic parsing to interpret user commands. A significant challenge in this domain is the limited vocabulary of training datasets compared to real-world usage. For example, the ATIS dataset contains fewer than 1,000 words, whereas domain-specific reviews can exceed 100,000 words. When users employ synonyms or new terminology (OOV words), standard parsers often fail. Furthermore, fine-tuning models on new data frequently degrades performance on original tasks. This research addresses the dual goals of runtime robustness to OOV words and stable continual learning.

Methodology

The researchers propose three primary enhancements to standard Sequence-to-Sequence (Seq2Seq) and Sequence-to-Tree models:

Robust Embeddings

Instead of random initialization for unknown tokens, the models utilize pre-trained word embeddings (e.g., GloVe). This enables the parser to leverage semantic similarities between OOV words (like "large") and known vocabulary (like "big").

Argument Transfer Model

The Argument Transfer decoder incorporates a neural classifier to predict when an "argument" (e.g., a city name or date) should be output. Upon detection, a binary lookup vector restricts output choices to specific arguments present in the input query, preventing confusion from unknown entities.

Comparison of (a) standard seq2seq+attention decoder and (b) the proposed argument transfer decoder.

Hybrid Paraphrase Generation

To handle OOV-heavy queries at runtime, a modular "translator" is placed before the parser. If the base parser's confidence score falls below a threshold, the query is routed to a paraphrase generator. This component converts OOV terms into the parser's known vocabulary before final processing.

Workflow of the hybrid paraphrase generation system for handling low-confidence queries.

Experimental Setup

The models were evaluated on three benchmark datasets: GEO (US geography), JOB (job listings), and ATIS (airline reservations). To specifically test OOV performance, the authors developed PARA-GEO, PARA-JOB, and PARA-ATIS by substituting words in the original test sets with synonyms from WordNet.

Results and Discussion

The proposed models achieved state-of-the-art (SOTA) accuracy on standard benchmarks, reaching 88.3% on GEO and 85.27% on ATIS.

OOV Resilience

On the PARA-GEO dataset, the baseline model's accuracy was 53.6%. The combination of paraphrase generation and argument transfer significantly improved this to 65.61%.

Stability in Continual Learning

A key finding was the mitigation of the "fine-tuning trap." While baseline models often saw accuracy drops of up to 2% on original tasks after retraining for new words, the hybrid paraphrase model demonstrated remarkable stability. In the ATIS dataset, it achieved a 0.08% gain in overall accuracy after learning new terms, indicating effective continual learning without forgetting.

Conclusion

This work presents a robust framework for semantic parsing in the presence of OOV words. By combining pre-trained embeddings, constrained decoding via argument transfer, and modular paraphrasing, the authors demonstrate that intelligent agents can expand their vocabulary without sacrificing existing capabilities. Future research will explore more complex structural paraphrasing and long-term sequential learning.

Summary updated

Abstract

Semantic parsers play a vital role in intelligent agents to convert natural language instructions to an actionable logical form representation. However, after deployment, these parsers suffer from poor accuracy on encountering out-of-vocabulary (OOV) words, or significant accuracy drop on previously supported instructions after retraining. Achieving both goals simultaneously is non-trivial. In this paper, we propose novel neural networks based parsers to learn OOV words; one incorporating a new hybrid paraphrase generation model, and an enhanced sequence-to-sequence model. Extensive experiments on both benchmark and custom datasets show our new parsers achieve significant accuracy gain on OOV words and phrases, and in the meanwhile learn OOV words while maintaining accuracy on previously supported instructions.
BibTeX
@inproceedings{ray2018learning,
  author = {Avik Ray and Yilin Shen and Hongxia Jin},
  title = {{Learning Out-of-Vocabulary Words in Intelligent Personal Agents}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2018},
  url = {https://openreview.net/forum?id=B1bEJ7MuWr},
}
Figure