On Collaborating Small and Large Models For Few-shot Intent Detection

Peng Chen, Bang WangEMNLP 2025

TL;DR

This paper introduces FCSLM, a novel framework that combines a small, efficient prediction model (PM) with a powerful large language model (LLM) to tackle the challenging problem of few-shot intent detection. By intelligently collaborating these models during both training and inference, FCSLM achieves state-of-the-art performance while significantly reducing computational costs.

What Problem Does This Paper Address?

Intent detection is crucial for dialogue systems, aiming to understand user queries and classify them into predefined categories or identify them as out-of-scope (OOS). However, in real-world scenarios, obtaining labeled data is expensive, leading to the "few-shot intent detection" (FSID) problem, where models must perform well with very limited examples. Existing approaches face trade-offs: smaller, pre-trained language models (PLMs) are efficient but struggle with nuanced semantic understanding, especially for OOS detection. Large language models (LLMs) excel at semantic reasoning but suffer from high inference costs and can be confused by similar intents, particularly in OOS scenarios. This paper addresses the need for a method that leverages the strengths of both small and large models to achieve high accuracy and efficiency in FSID.

Methods

FCSLM employs a two-phase approach:

  • Offline Training Phase:
    • Self-supervised Pretraining (SSP): An off-the-shelf PLM (like RoBERTa) is first pre-trained using a contrastive loss and masked language modeling (MLM) on a large public dataset. This enhances its understanding of semantic nuances.
  • Supervised Fine-tuning (SFT): The pre-trained PLM is fine-tuned on the FSID task dataset. Crucially, LLMs are used for data augmentation here. This includes:
  • Query Augmentation with Trimming (QAT): LLMs generate paraphrased queries to enrich the training data. A trimming mechanism filters out generated queries that drift semantically.
  • Label Augmentation (LA): LLMs generate more descriptive labels for intents to improve clarity.
  • Multi-Predictor Sampling (MPS): To mitigate prediction uncertainty from a single model, multiple "prediction heads" are trained, acting as Bayesian linear classifiers.
  • Online Reasoning Phase (OCRP): This phase uses a multi-round process:
  • Round 1 (PM Prediction): The trained PM (fine-tuned PLM with MPS) makes an initial prediction. If confidence is high (based on a multi-head voting mechanism), the prediction is finalized.
  • Round 2 (LLM Collaboration): If the PM's prediction is uncertain, an LLM is invoked. It's prompted with the query and a curated list of candidate intents (from the PM's top-K predictions) along with their descriptions. The LLM selects the most suitable intent.
  • Round 3 (LLM Comparative Prediction): If the LLM's choice differs from the PM's top prediction, a final comparative round is performed where the LLM makes a more nuanced decision, considering semantic evidence.
  • Out-of-Scope (OOS) Detection: A plug-in module integrates the null intent into the second round prediction, allowing the LLM to explicitly choose OOS if none of the candidate intents are suitable.

Datasets

The paper uses three standard datasets for evaluation:

  • BANKING77: Focuses on banking-related intents.
  • CLINC150: A multi-domain dataset with various intent categories.
  • HWU64: A dataset for voice assistant intents.

For pre-training, a public dataset (SNIPS) is mentioned, and the paper utilizes data from these benchmark datasets for fine-tuning and evaluation in k-shot settings (0, 5, and 10 shots).

Key Insights and Ideas

  • Synergy of Small and Large Models: The core idea is that small models are good for high-confidence, quick predictions, while LLMs are best for resolving complex, uncertain cases. This collaboration is key to balancing performance and efficiency.
  • LLM-Augmented Training: Using LLMs not just for inference but also for generating high-quality training data (queries and labels) significantly boosts the performance of the smaller PM.
  • Multi-Round Reasoning for Confidence: The staged reasoning process allows the system to progressively refine its decision, starting with a fast PM and only engaging the more computationally expensive LLM when necessary.
  • Top-K Candidate Selection: Limiting the LLM's choices to a subset of top candidates from the PM (rather than all intents) drastically reduces LLM inference cost and improves accuracy by mitigating prompt complexity.
  • OOS Detection as a Null Intent: Treating OOS as a distinct "null" intent within the reasoning process, rather than a separate classification step, is more effective, especially in low-resource settings.

Key Results

  • FCSLM achieves state-of-the-art performance on all three benchmark datasets (BANKING77, CLINC150, HWU64) for both in-scope intent classification and OOS detection across 0/5/10-shot settings.
  • For example, on BANKING77 in the 5-shot setting, FCSLM achieved 87.66% accuracy, outperforming competitors.
  • The framework significantly reduces LLM calls and token overhead, leading to substantial efficiency gains compared to pure LLM approaches. In one experiment, LLM invocation rate dropped to 31.92% while achieving higher accuracy than a full LLM baseline.
  • Ablation studies confirm the importance of each component: removing LLM-based augmentation or the multi-round reasoning significantly degrades performance.

Limitations and Future Work

  • Limitations:
    • In zero-shot scenarios, the lack of real few-shot queries can hinder the LLM's ability to generate semantically grounded training data for the PM, potentially limiting its ability to learn fine-grained distinctions.
  • The performance of FCSLM can be sensitive to the choice of top-K candidate intents, though a moderate range (10%-50%) generally yields good results.
  • Future Work:
    • Further improve OOS detection by exploring deeper collaboration between small and large models.
  • Investigate methods to further enhance in-scope intent classification performance.
  • The paper demonstrates FCSLM's compatibility with LLMs of varying sizes, suggesting potential for deployment with different LLM resources.

Summary updated

BibTeX
@inproceedings{chen2025collaborating,
  author = {Peng Chen and Bang Wang},
  title = {{On Collaborating Small and Large Models For Few-shot Intent Detection}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure