TL;DR
This paper introduces PICD-Instruct, a novel framework for few-shot multi-intent spoken language understanding (SLU) that leverages generative instruction learning. It addresses key challenges like slot-label mismatches and relational confusion in multi-intent scenarios, achieving state-of-the-art results on benchmark datasets.
What Problem Does This Paper Address?
Spoken Language Understanding (SLU) is crucial for task-oriented dialogue systems, enabling them to interpret user commands. A particularly challenging aspect is multi-intent SLU, where a single utterance can express multiple user goals (e.g., "Book a flight and check the weather"). The core difficulty lies in accurately identifying all intents and extracting the relevant information (slots) for each, especially when dealing with few-shot scenarios – situations where only a tiny amount of labeled data is available for training.
Traditional approaches struggle in few-shot multi-intent SLU due to several issues:
- Slot-label mismatches: Large Language Models (LLMs) often generate a different number of slots than expected, leading to errors.
- Relational confusion: Understanding the dependencies between multiple intents and their associated slots is complex, especially with limited data.
- Intent count uncertainty: The number of intents in an utterance can vary, making it hard for models to consistently detect them.
This paper aims to overcome these limitations by proposing a new instruction learning framework for LLMs specifically designed for few-shot multi-intent SLU.
Methods
The authors propose PICD-Instruct, a generative framework built upon three types of instructions designed to guide LLMs:
- Basic Instructions (BI): This instruction type shifts from token-level slot labeling to an entity-word pairing approach. Instead of assigning a label to each word, BI guides the LLM to identify entities and the specific words that constitute them. This helps mitigate the mismatch between the number of generated slots and the utterance length, a common LLM challenge.
- Pairwise Interaction Instructions (PII): To explicitly model the bidirectional dependencies between intents and slots, PII guides the LLM to pair each identified intent with its corresponding entities. This helps reduce confusion in multi-intent scenarios by establishing clear relationships.
- Contrastive Distinct Instructions (CDI): This instruction type addresses the challenge of varying intent counts. CDI trains the LLM to determine whether two utterances contain the same number of intents by using contrastive learning. This enhances the model's ability to perceive and correctly count intents within an utterance.
The framework formulates these instructions in a question-answer format, leveraging the LLM's generative capabilities.
Datasets
The authors evaluate their proposed method on two few-shot multi-intent SLU datasets:
- FewShotMixATIS
- FewShotMixSNIPS
These datasets are derived from existing MixATIS and MixSNIPS datasets using a dynamic sampling algorithm to create few-shot scenarios (ranging from 2-shot to 10-shot for training). The test sets are designed to contain more slot types than the training sets to better reflect real-world generalization capabilities.
Key Insights and Ideas
- Instruction Tuning for Nuance: The paper highlights that carefully crafted instructions can guide LLMs to perform complex tasks like few-shot multi-intent SLU with remarkable accuracy, going beyond simple prompt engineering.
- Entity-Word Pairing Solves Slot Mismatch: The BI instruction's focus on entity-word correspondence, rather than token-level labeling, is a clever way to bypass a fundamental LLM limitation in generative SLU.
- Contrastive Learning for Intent Counting: Using contrastive learning to teach the LLM to compare intent counts between utterances is an innovative approach to handle the inherent uncertainty in multi-intent scenarios.
- Synergy of Instructions: The combination of BI, PII, and CDI is crucial. Each instruction type addresses a specific challenge, and their integration leads to a synergistic improvement in overall performance.
Key Results
PICD-Instruct achieved state-of-the-art (SOTA) performance on both the FewShotMixATIS and FewShotMixSNIPS datasets across various few-shot settings. For instance, in the 2-shot setting on FewShotMixATIS, PICD-Instruct surpassed BERT-SIF by significant margins in intent accuracy, slot F1, and overall accuracy. The model also demonstrated strong generalization capabilities, transferring from a single-domain to a multi-domain dataset. Ablation studies confirmed the effectiveness of each component (BI, PII, CDI), showing that removing any one of them led to performance degradation.
Limitations and Future Work
- Annotation Noise: The authors acknowledge that using GPT-4o for annotating pairwise interaction instructions introduced some noise, as GPT-40 occasionally omitted entities. Despite this, their model showed robustness.
- External Knowledge Integration: The paper suggests that integrating external knowledge about intent and slot labels could further enhance LLMs' comprehension and mitigate hallucination issues. This is identified as a promising direction for future work.
- Model Size vs. Performance: While larger models generally perform better, the gains diminish after a certain point, suggesting that for resource-constrained scenarios, smaller, well-instructed models can be highly effective.
Summary updated