TL;DR
IntentGPT introduces a training-free framework that leverages frozen Large Language Models (such as GPT-4 and Llama-2) to discover and classify user intents in conversational systems. By pairing meta-prompt generation with dynamic few-shot retrieval and an evolving memory of discovered intents, it outperforms traditional multi-stage, fine-tuned pipelines while using only a fraction of labeled data.
What Problem Does This Paper Address?
In conversational AI, dialogue systems must map user utterances to specific goals, known as Intent Detection. In real-world applications, however, user goals constantly evolve. While standard intent detection operates in a closed-world setup (classifying among fixed categories or routing out-of-scope queries to a catch-all "other" bucket), Intent Discovery requires an open-world setup: recognizing known intents while dynamically uncovering and naming novel intents on the fly.
Prior intent discovery methods rely heavily on multi-stage training—such as pretraining sentence encoders, fine-tuning via contrastive objectives, and applying complex clustering algorithms. These approaches demand extensive domain-specific labeled data and significant computational overhead. IntentGPT asks: Can we eliminate model training altogether by framing intent discovery as an in-context learning (ICL) problem with frozen LLMs?
Methods
IntentGPT relies entirely on prompting frozen LLMs without updating model weights. Its architecture centers around four key components:
- In-Context Prompt Generator (ICPG): Before inference, GPT-4 is given a tiny sample of known intents ($x=2$ examples per intent) to automatically write a tailored, domain-specific instruction prompt for the task. This prompt establishes guidelines, reasoning structure, and explicit instructions never to output generic labels like
"unknown". - Semantic Few-Shot Sampler (SFS): To supply relevant context demonstrations, a SentenceBERT (SBERT) encoder embeds test queries and retrieves the most semantically similar examples from a 10% labeled pool using $k$-Nearest Neighbors ($k$NN).
- Known Intent Feedback (KIF & SKIF): As the model discovers new intents during inference, they are appended to an active database. At each test iteration, known and newly discovered intents are injected into the prompt so the LLM can reuse them rather than fragmenting categories. When context length is tight, Semantic KIF (SKIF) uses $k$NN to inject only the top-$k$ most relevant intent candidates.
- Intent Predictor & Clustering: A frozen LLM predicts intents for test batches. For standard benchmark evaluation, predicted intent strings are embedded via SBERT, the cluster count $K$ is automatically estimated using DBSCAN ($\epsilon=0.5$), and K-Means clustering is performed.
[Training Data Sample] -> [In-Context Prompt Generator]
|
v
[Test Batch] + [Semantic Few-Shot Sampler] + [Known Intent Feedback]
|
v
[Frozen Intent Predictor LLM]
|
+---------------+---------------+
v v
[Discovered Intents] [Intent Predictions]
(updates feedback loop) |
v
[SBERT + K-Means]
Datasets
The authors evaluate on two widely used intent discovery benchmarks under various Known Intent Ratios (KIR $\in \{0.25, 0.50, 0.75\}$):
- CLINC: 150 unique intents spanning 10 diverse domains (travel, banking, work, etc.), containing 18,000 training, 2,250 validation, and 2,250 test utterances.
- BANKING: A fine-grained, finance-specific dataset with 77 intents, comprising 9,003 training, 1,000 validation, and 3,080 test utterances.
No new datasets were introduced.
Key Insights and Ideas
- Feedback Loops Enable Lifelong In-Context Learning: LLMs natively generate novel class names, but without memory they continuously coin redundant synonyms. The Known Intent Feedback (KIF) loop transforms the static LLM into an active agent that builds an internal registry of newly discovered classes.
- LLMs Write Better Prompts Than Humans: Ablation results demonstrate that letting GPT-4 generate domain-specific instructions (ICP) significantly improves intent assignment compared to human-written templates.
- Semantic Retrieval Outperforms Random Demonstration: Retrieving context demonstrations that are close in SBERT embedding space (SFS) yields consistent clustering gains over arbitrary few-shot selection.
- Generalization vs. Memorization: To verify that GPT-4 is not simply regurgitating memorized test labels, the authors introduce Fréchet BERT Distance (FBD) to measure distribution overlap between discovered and ground-truth label sets. GPT-4 scored an FBD of ~0.54–0.55 on both benchmarks, indicating that it generated semantically coherent, novel descriptors (e.g., predicting
device_pairinginstead of ground-truthsync_device) rather than exact memorized strings.
Key Results
On the standard benchmark configuration ($\text{KIR} = 0.75$):
- CLINC: IntentGPT-4 (50-shot) achieved 96.06% NMI, 84.76% ARI, and 88.76% ACC, beating top semi-supervised methods such as SCL (94.75 NMI / 81.64 ARI) and DeepAligned (94.03 NMI / 79.82 ARI).
- BANKING: IntentGPT-4 (50-shot) established new state-of-the-art results across all metrics with 85.94% NMI, 66.66% ARI, and 77.21% ACC (outperforming SCL's 85.04 NMI / 65.43 ARI / 76.55 ACC).
- Unsupervised / 0-Shot: Even without demonstrations (0-shot with KIF), IntentGPT-4 outperformed all prior unsupervised clustering techniques, scoring 94.35 NMI on CLINC and 81.42 NMI on BANKING.
Limitations and Future Work
- Context Constraints and API Costs: Querying commercial models at scale incurs substantial API latency and expense (~$7 to $9.65 and up to 2 hours per test evaluation on GPT-4). Open-source models like Llama-2 reduce costs but lag behind in accuracy.
- Sensitivity to Temperature: As sampling temperature increases, clustering accuracy and ARI degrade rapidly due to stochastic drift in intent naming.
- Low-KIR Bottlenecks: When the initial proportion of known intents is very small ($\text{KIR} = 0.25$), adding more few-shot demonstrations yields diminishing returns, as the lack of initial structure can cause open-ended generation to drift.
- Future Directions: Incorporating context-compression techniques, mitigating latency, and exploring privacy-preserving local LLM deployments.
Summary updated