RetroNLU: Retrieval Augmented Task-Oriented Semantic Parsing

Vivek Gupta, Akshat Shrivastava, Adithya Sagar, Armen Aghajanyan, Denis Savenkov

TL;DR

RETRONLU enhances sequence-to-sequence models for task-oriented semantic parsing by retrieving nearest-neighbor examples (either raw utterances or their semantic parses) from an external memory index and appending them as input context. This retrieval-augmented strategy improves frame accuracy across domains and is remarkably data-efficient, matching baseline accuracy trained on 100% of data using just 40% of the training examples.

What Problem Does This Paper Address?

Task-oriented conversational assistants rely on semantic parsing to convert user queries into structured representations—such as hierarchical intents and slot arguments (e.g., mapping "please add 20 minutes on the lasagna timer" into intent and slot tags). While pre-trained language models store substantial world knowledge in their weights (parametric memory), relying solely on parameters causes known issues:

  1. High sample complexity: They require large annotated datasets to capture long-tail or complex compositional structures.
  2. Difficult maintenance: Updating or correcting misclassifications in production requires retraining the entire network.
  3. Hallucinations and poor tail performance: Models struggle on rare queries that appear infrequently during training.

The authors explore whether augmenting a seq2seq parser with an explicit, non-parametric memory index can boost performance, improve data efficiency, and support semi-supervised training.

Methods

RETRONLU introduces an architecture-agnostic retrieval pipeline on top of a standard sequence-to-sequence parser with a copy mechanism:

  1. Indexing: Utterances in the training set are encoded into sentence embeddings via pre-trained BART (averaging token embeddings from the final layer). These vectors are indexed using FAISS for cosine similarity search.
  2. Retrieval: Given an input utterance at train or inference time, the model queries the index to retrieve the top-$k$ most similar training examples (excluding exact matches during training).
  3. Augmentation: The input is augmented by prepending the retrieved context to the original utterance using a delimiter token (|). Two settings are evaluated:
  4. utterance-nn: Appends the retrieved unlabeled utterance text (useful for semi-supervised setups).
  5. semparse-nn: Appends the ground-truth semantic parse of the retrieved neighbor.
  6. Parsing: A BART-based Seq2Seq-CopyPtr model takes the augmented sequence as input and autoregressively generates the decoupled canonical semantic parse, using its pointer-generator copy mechanism to pull tokens directly from the input or the retrieved exemplars.
[Retrieved Semparse or Utterance] | [Input Utterance]  ---> [Seq2Seq-CopyPtr] ---> [Predicted Semparse]

Datasets

  • TOPv2 (Task-Oriented Parsing v2): The primary benchmark used, featuring complex and compositional hierarchical queries across multiple domains. The experiments focus on six source domains: alarm, event, messaging, music, navigation, and timer.
  • No new benchmarks were introduced; existing canonical splits and decoupled representations were used.

Key Insights and Ideas

  • Exemplars act as structural templates: Prepending semantic parses allows the copy mechanism to copy structural skeleton tokens (intents and slot names), freeing the model from needing to generate complex tree structures entirely from parametric weights.
  • Unlabeled text still helps: Even without retrieving ground-truth parses (utterance-nn), appending similar raw utterances improved accuracy over the non-retrieval baseline, likely via memorization-based generalization.
  • Disproportionate gains on the long tail: While both simple and complex queries benefit, relative improvements are starkest on rare frames (a 4.11% relative gain on the lowest-frequency frames vs. 1.11% on high-frequency frames).
  • Graceful degradation under noise: When fed cross-domain or random neighbors from the top-100 search, the model still matched or slightly exceeded the non-retrieval baseline. This indicates that the attention and copy mechanisms learn when to attend to external context and when to fall back on internal parametric knowledge.

Key Results

  • Full-Data Performance: On the complete TOPv2 source dataset, semparse-nn achieves 85.74% micro-average frame accuracy (+1.31% over the 84.43% baseline) and 85.82% macro-average (+1.16%).
  • Data Efficiency: Under limited supervision, retrieval provides dramatic gains:
  • At 10% training data, semparse-nn achieves 78.43% vs. the baseline's 74.19% (+4.24% absolute).
  • With only 40% of labeled training data, RETRONLU matches the performance of the baseline trained on 100% of the data.
  • Complex Utterances: On compositional frames (depth $\ge 2$), semparse-nn achieved a ~2% absolute improvement over baseline.
  • Diminishing Returns with $k$: Increasing from $k=1$ to $k=3$ neighbors offered marginal improvements ($85.74\% \to 85.80\%$) due to frame redundancy and context window limitations.

Limitations and Future Work

  • Static embeddings / Decoupled training: Retrieval uses fixed, off-the-shelf BART sentence embeddings. The retriever is not trained end-to-end with the parsing objective.
  • Context length overhead: Prepending multiple exemplars increases sequence length, running into token length constraints (e.g., BART’s 512-token limit) and slowing down inference.
  • Future Directions:
  • Joint end-to-end training of the retrieval encoder and parser.
  • Applying cross-lingual embeddings (such as mBART) to enable cross-lingual retrieval-augmented parsing.
  • Investigating few-shot and zero-shot domain adaptation setups.

Summary updated

Abstract

While large pre-trained language models accumulate a lot of knowledge in their parameters, it has been demonstrated that augmenting it with non-parametric retrieval-based memory has a number of benefits from accuracy improvements to data efficiency for knowledge-focused tasks, such as question answering. In this paper, we are applying retrieval-based modeling ideas to the problem of multi-domain task-oriented semantic parsing for conversational assistants. Our approach, RetroNLU, extends a sequence-to-sequence model architecture with a retrieval component, used to fetch existing similar examples and provide them as an additional input to the model. In particular, we analyze two settings, where we augment an input with (a) retrieved nearest neighbor utterances (utterance-nn), and (b) ground-truth semantic parses of nearest neighbor utterances (semparse-nn). Our technique outperforms the baseline method by 1.5% absolute macro-F1, especially at the low resource setting, matching the baseline model accuracy with only 40% of the data. Furthermore, we analyze the nearest neighbor retrieval component's quality, model sensitivity and break down the performance for semantic parses of different utterance complexity.
BibTeX
@inproceedings{gupta2022retronlu,
  author = {Vivek Gupta and Akshat Shrivastava and Adithya Sagar and Armen Aghajanyan and Denis Savenkov},
  title = {{RetroNLU: Retrieval Augmented Task-Oriented Semantic Parsing}},
  booktitle = {ConvAI @ ACL},
  year = {2022},
  url = {https://openreview.net/forum?id=A8safECdddI},
}

Loading related content...

Figure