RETRONLU: Retrieval Augmented Task-Oriented Semantic Parsing

Vivek Gupta, Akshat Shrivastava, Adithya Sagar, Armen Aghajanyan, Denis Savenkov

Abstract

While large pre-trained language models accumulate a lot of knowledge in their parameters, it has been demonstrated that augmenting it with non-parametric retrieval-based memory has a number of benefits from accuracy improvements to data efficiency for knowledge-focused tasks, such as question answering. In this paper, we are applying retrieval-based modeling ideas to the problem of multi-domain task-oriented semantic parsing for conversational assistants. Our approach, RetroNLU, extends a sequence-to-sequence model architecture with a retrieval component, used to fetch existing similar examples and provide them as an additional input to the model. In particular, we analyze two settings, where we augment an input with (a) retrieved nearest neighbor utterances (utterance-nn), and (b) ground-truth semantic parses of nearest neighbor utterances (semparse-nn). Our technique outperforms the baseline method by 1.5% absolute macro-F1, especially at the low resource setting, matching the baseline model accuracy with only 40% of the data.

Introduction

In conversational AI, semantic parsing serves as the critical bridge between human language and machine-executable actions. This process converts user requests—such as "Set an alarm for 7 AM"—into structured formats. While modern Large Language Models (LLMs) excel at this task, they primarily rely on parametric knowledge stored within their weights. This reliance introduces several challenges: rigidity in updating knowledge, difficulty handling the "long tail" of rare or complex hierarchical commands, and a high demand for expensive labeled training data.

To address these limitations, we propose RETRONLU, a retrieval-augmented approach that utilizes non-parametric memory. By maintaining an external index of examples, the model can generalize more effectively and learn with significantly higher data efficiency.

The RetroNLU Framework

The RetroNLU architecture: Indexing, Retrieval, Augmentation, and Semantic Parsing.

The RETRONLU framework operates through a four-stage pipeline:

  1. Indexing: We utilize a pre-trained BART model to generate embeddings for all training utterances, which are then stored in a fast retrieval index using the FAISS library.
  2. Retrieval: For every new query, the system identifies the "nearest neighbor" (the most semantically similar example) from the index.
  3. Augmentation: The retrieved neighbor—either the raw text (utterance-nn) or the text paired with its structured semantic parse (semparse-nn)—is appended to the user input using a special separator token.
  4. Semantic Parsing: This augmented input is processed by a BART-based Seq2Seq model. A key feature is the copy mechanism, which allows the model to either generate new tokens or directly copy structural templates and labels from the retrieved example.

Experimental Results

We evaluated RetroNLU using the TOPv2 (Task-Oriented Parsing v2) dataset, which spans eight domains and includes both simple "flat" frames and complex "compositional" nested structures.

Data Efficiency and Performance

One of the most significant advantages of RetroNLU is its extreme data efficiency. Our results demonstrate that the retrieval-augmented model can match the performance of a 100% trained baseline using only 40% of the training data. In extreme low-resource settings (10% data), RetroNLU provides a massive 4.24% improvement over standard models.

Frame accuracy vs. training data percentage, highlighting RetroNLU's superior performance in low-resource settings.

Handling Complexity and the Long Tail

RetroNLU acts as a structural "cheat sheet." The copy mechanism is particularly effective for complex, nested queries where models typically struggle to maintain hierarchy. For these complex frames, the semparse-nn variant improved performance by 2% over the baseline. Furthermore, the gains were most pronounced for rare queries (the "long tail"), where we observed a relative improvement of nearly 4% compared to approximately 1% for common queries.

Conclusion

RETRONLU demonstrates that augmenting parametric models with non-parametric retrieval significantly enhances task-oriented semantic parsing. The model is remarkably robust; even when provided with suboptimal neighbors, it learns to weigh its internal knowledge against the retrieved context. Future work will explore joint training of the retrieval and parsing components, as well as extending these benefits to cross-lingual applications.

Summary updated

Abstract

While large pre-trained language models accumulate a lot of knowledge in their parameters, it has been demonstrated that augmenting it with non-parametric retrieval-based memory has a number of benefits from accuracy improvements to data efficiency for knowledge-focused tasks, such as question answering. In this paper, we are applying retrieval-based modeling ideas to the problem of multi-domain task-oriented semantic parsing for conversational assistants. Our approach, RetroNLU, extends a sequence-to-sequence model architecture with a retrieval component, used to fetch existing similar examples and provide them as an additional input to the model. In particular, we analyze two settings, where we augment an input with (a) retrieved nearest neighbor utterances (utterance-nn), and (b) ground-truth semantic parses of nearest neighbor utterances (semparse-nn). Our technique outperforms the baseline method by 1.5% absolute macro-F1, especially at the low resource setting, matching the baseline model accuracy with only 40% of the data. Furthermore, we analyze the nearest neighbor retrieval component's quality, model sensitivity and break down the performance for semantic parses of different utterance complexity.
BibTeX
@inproceedings{gupta2022retronlu,
  author = {Vivek Gupta and Akshat Shrivastava and Adithya Sagar and Armen Aghajanyan and Denis Savenkov},
  title = {{RETRONLU: Retrieval Augmented Task-Oriented Semantic Parsing}},
  booktitle = {ConvAI @ ACL},
  year = {2022},
  url = {https://openreview.net/forum?id=A8safECdddI},
}
Figure