Generalized Multiple Intent Conditioned Slot Filling

Harshil Shah, Arthur Wilcke, Marius Cobzarenco, Cristi Cobzarenco, Edward Challis, David BarberarXiv 2023 · OpenReview

TL;DR

This paper tackles the limitation of traditional slot filling models that assume a single intent per message. They reframe slot filling as a JSON generation task, enabling models to handle multiple, even repeated, intents within a single message. This generalization is achieved by training large language models on a combination of pre-training and domain-specific datasets, leading to improved performance and generalization capabilities.

What Problem Does This Paper Address?

Natural Language Understanding (NLU) systems often break down user requests into intents (what the user wants to do) and slots (the specific details or entities related to that intent). A common assumption in traditional slot filling is that each message expresses only one intent, and therefore, each entity slot can only belong to that single intent. This assumption is problematic in real-world scenarios, like customer service emails or chat logs, where a single message can contain multiple distinct requests or even multiple instances of the same request. For example, a user might want to both increase the quantity of one item and remove another item from an order, all within the same message. Existing methods struggle with this, leading to incomplete or inaccurate understanding. This paper aims to overcome this limitation by developing a slot filling approach that can handle messages with multiple, potentially repeated, intents.

Methods

The core innovation of this work is reframing the slot filling problem as a text generation task, specifically generating a structured JSON output. Instead of traditional sequence tagging, they leverage large language models (LLMs) to directly output a JSON string that encapsulates all identified intents and their corresponding entities.

Their approach involves:

  • Prompt Engineering: They construct prompts for the LLM that include the instruction, the list of possible intents and their associated entities, and the user's message.
  • Zero-Shot and One-Shot Learning: They experiment with both zero-shot prompting (providing only the instruction and the message) and one-shot prompting (including a single example of a correctly formatted JSON output in the prompt).
  • Model Choice: They utilize the Flan-T5 model, a variant of T5 known for its instruction-following capabilities, due to its encoder-decoder architecture and strong performance with fewer parameters.

Datasets

The authors created and utilized several datasets:

  • Pre-training Dataset: This dataset was constructed by combining:
  • DBpedia: Abstracts and infobox properties from Wikipedia articles, where article "types" were used as a proxy for intents.
  • Existing Slot Filling Datasets: Datasets like ATIS, SNIPS, and SLURP were converted into the JSON generation format. Importantly, they did not enforce the unique intent restriction when combining utterances from these datasets.
  • In-domain Dataset: A synthetic dataset of over 10,000 email-JSON pairs was generated using GPT-3. This dataset was specifically designed to mimic real-world communications data with diverse intents.

Key Insights and Ideas

  • JSON Generation as a Unified Framework: Treating slot filling as a JSON generation task is a powerful way to handle complex, multi-intent messages, moving beyond the limitations of sequence tagging.
  • LLMs Excel at Structured Output: LLMs, particularly instruction-tuned ones like Flan-T5, can be effectively trained to produce structured outputs like JSON, even when dealing with nuanced semantic information.
  • Pre-training is Crucial for Downstream Tasks: Even though the pre-training data (DBpedia) is semantically different from the target communications data, it significantly improves the model's ability to extract relevant entities and format them as JSON, making subsequent in-domain fine-tuning much more effective.
  • One-Shot Prompting Isn't Always Better: Surprisingly, for in-domain evaluation data (which is from the same distribution as training), zero-shot prompting sometimes outperformed one-shot prompting. This suggests that when the model is well-trained on the target distribution, explicit examples might not be as critical.
  • Generalization to Unseen Intents: The model demonstrates a notable ability to generalize to intent types that were not present in its training data, a critical capability for real-world NLU systems.

Key Results

  • Improved Performance: Both pre-training and in-domain training significantly improved the model's performance on the generalized slot filling task.
  • High F1 Scores: On the in-domain test set, the model achieved high F1 scores, reaching 93.42% object level and 95.62% key-value level with zero-shot prompting after both pre-training and in-domain training.
  • Generalization: When tested on intents not seen during training (specifically, moving "Payroll >..." intents to the test set), the model still performed well, achieving 84.98% object level F1 and 90.65% key-value level F1 with zero-shot prompting.
  • Valid JSON Output: The trained model consistently outputs valid JSON and avoids hallucinating irrelevant information.

Limitations and Future Work

  • Computational Cost: Training large language models is computationally expensive, with one-shot prompted models requiring significant GPU hours. However, the paper notes that domain-specific fine-tuning is relatively fast after initial pre-training.
  • Hallucination Risk: While mitigated by training on existing text and careful prompting, the risk of generating nonsensical or irrelevant text (hallucination) is inherent in generative models. The authors suggest post-processing as a safeguard.
  • Synthetic Data Reliance: Due to the privacy of communications data, the in-domain dataset was synthetic. While the authors hope for good generalization to real data, further fine-tuning might be necessary for models deployed on actual user data from different distributions.
  • Future Work: The paper suggests that their pre-training dataset could be valuable for other entity extraction and grouping tasks. They also highlight the broader applicability of training LLMs to produce structured outputs for downstream API consumption.

Summary updated

Abstract

Natural language understanding includes the tasks of intent detection (identifying a user's objectives) and slot filling (extracting the entities relevant to those objectives). Prior slot filling methods assume that each intent type cannot occur more than once within a message, however this is often not a valid assumption for real-world settings. In this work, we generalize slot filling by removing the constraint of unique intents in a message. We cast this as a JSON generation task and approach it using a language model. We create a pre-training dataset by combining DBpedia and existing slot filling datasets that we convert for JSON generation. We also generate an in-domain dataset using GPT-3. We train T5 models for this task (with and without exemplars in the prompt) and find that both training datasets improve performance, and that the model is able to generalize to intent types not seen during training.
BibTeX
@misc{shah2023generalized,
  author = {Harshil Shah and Arthur Wilcke and Marius Cobzarenco and Cristi Cobzarenco and Edward Challis and David Barber},
  title = {{Generalized Multiple Intent Conditioned Slot Filling}},
  year = {2023},
  eprint = {2305.11023},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.11023},
}
Figure