Conformal Intent Classification and Clarification for Fast and Accurate Intent Recognition

Floris den Hengst, Ralf Wolter, Patrick Altmeyer, Arda Kaygan

Introduction

In modern task-oriented dialogue systems, such as customer service chatbots, Intent Classification (IC) serves as the critical component that maps user utterances to specific actions (e.g., "lost card" or "check balance"). However, human language is inherently ambiguous, incomplete, or occasionally "out-of-scope" (OOS). Traditional IC systems typically face a binary choice when encountering uncertainty: they either provide a "top-1" guess that risks being incorrect or issue a generic "I didn't understand" response. Neither approach is optimal for user experience.

To address this, researchers have introduced CICC (Conformal Intent Classification and Clarification). This framework utilizes conformal prediction to transform a model's heuristic guess into a statistically guaranteed set of likely intents. By presenting these options through a concise clarification question, dialogue systems can resolve ambiguity more efficiently while maintaining a pre-defined confidence level.

The CICC Framework

The core of CICC is Split Conformal Prediction, a method that provides rigorous statistical guarantees for classification. Unlike standard classifiers that output a single label, CICC produces a prediction set.

System Architecture

The CICC pipeline integrates seamlessly with existing intent classifiers. As shown in the architecture diagram below, the system processes user input through a standard classifier, applies conformal prediction to generate a set of candidate intents, and then uses a generative Large Language Model (LLM) to formulate a natural clarification question if needed.

Figure 1: Overview of the CICC framework architecture, showing the flow from user input to clarification question generation.

The CICC Algorithm

The framework operates in three distinct stages based on the size of the prediction set:

  1. Direct Execution: If the prediction set contains exactly one intent, the system proceeds with that intent.
  2. Clarification: If the set contains a small number of intents (below a threshold $th$, typically 7), a generative LLM (e.g., Vicuna-7b) phrases a natural clarification question, such as "Did you lose your card or was it stolen?".
  3. Rejection/OOS Detection: If the set is too large, the input is deemed too ambiguous or out-of-scope and is rejected.

A significant advantage of CICC is its model-agnostic nature; it functions as a post-processing layer and does not require retraining the underlying intent classifier.

Experimental Evaluation

The framework was rigorously tested across seven diverse intent recognition datasets, including public benchmarks like ACID (insurance), ATIS (airline travel), B77 (banking), and HWU64 (home automation). It was also evaluated on IND, a proprietary Dutch-language dataset from ING Group, and specialized variants designed for Out-of-Scope (OOS) detection.

Key Results and Insights

The experimental results demonstrate that CICC effectively bridges the gap between heuristic uncertainty and statistical reliability.

Statistical Guarantees and Coverage

A primary finding is CICC's ability to strictly adhere to required confidence levels. As illustrated in the performance plots below, the framework maintains the target coverage (e.g., 98% for $\alpha=0.02$) across various datasets, whereas standard heuristic baselines often fail to include the true intent in their top-N predictions.

Figure 2: Performance metrics across datasets, showing average set size, SSC, and coverage relative to the target confidence level (1-alpha).

Efficiency and OOS Performance

CICC proved highly efficient, generating clarification questions with an average of only 2-3 intents. This minimizes the cognitive load on the user compared to broader, less certain baselines. Furthermore, a specialized version, CICC-OOS, achieved high F1-scores (up to 0.91) for detecting out-of-scope queries, outperforming several specialized OOS detection methods.

Conclusion and Future Work

CICC offers a robust solution for managing ambiguity in dialogue systems by providing mathematical guarantees on intent coverage. By leveraging LLMs for natural language generation, it ensures that statistical rigor does not come at the cost of conversational fluidity.

Future research directions include investigating "stopping rules" for multi-turn clarifications and conducting extensive human evaluations to assess the naturalness of LLM-generated questions. Additionally, addressing sensitivity to calibration data drift remains a key area for ensuring long-term system reliability.

Summary updated

Abstract

We present Conformal Intent Classification and Clarification (CICC), a framework for fast and accurate intent classification for task-oriented dialogue systems. The framework turns heuristic uncertainty scores of any intent classifier into a clarification question that is guaranteed to contain the true intent at a pre-defined confidence level.By disambiguating between a small number of likely intents, the user query can be resolved quickly and accurately. Additionally, we propose to augment the framework for out-of-scope detection.In a comparative evaluation using seven intent recognition datasets we find that CICC generates small clarification questions and is capable of out-of-scope detection.CICC can help practitioners and researchers substantially in improving the user experience of dialogue agents with specific clarification questions.
BibTeX
@inproceedings{hengst2024conformal,
  author = {Floris den Hengst and Ralf Wolter and Patrick Altmeyer and Arda Kaygan},
  title = {{Conformal Intent Classification and Clarification for Fast and Accurate Intent Recognition}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-naacl.156},
}
Figure