Abstract
We present Conformal Intent Classification and Clarification (CICC), a framework for fast and accurate intent classification for task-oriented dialogue systems. The framework turns heuristic uncertainty scores of any intent classifier into a clarification question that is guaranteed to contain the true intent at a pre-defined confidence level. By disambiguating between a small number of likely intents, the user query can be resolved quickly and accurately. Additionally, we propose to augment the framework for out-of-scope detection. In a comparative evaluation using seven intent recognition datasets we find that CICC generates small clarification questions and is capable of out-of-scope detection. CICC can help practitioners and researchers substantially in improving the user experience of dialogue agents with specific clarification questions.
Introduction
In modern task-oriented dialogue systems, such as banking or insurance bots, Intent Classification (IC) serves as the critical first step in understanding user needs. However, real-world user input is frequently messy, containing typos, incomplete thoughts, or linguistic ambiguities. Most current systems handle this uncertainty through a "forced choice" approach: they either pick the single top-scoring intent (risking a wrong answer), provide a generic "I didn't understand" fallback (frustrating the user), or use arbitrary heuristic thresholds to ask clarification questions.
These methods lack rigorous uncertainty quantification. Developers currently lack a principled way to determine when a clarification question (CQ) is necessary, which intents should be included, and how to guarantee that the user's actual intent is among the options presented. The Conformal Intent Classification and Clarification (CICC) framework addresses these gaps by providing statistical guarantees for intent recognition.
The CICC Framework
CICC leverages Conformal Prediction (CP), a model-agnostic framework that transforms standard classifier outputs into prediction sets. Unlike traditional classifiers that output a single label, CICC produces a set of likely intents that is mathematically guaranteed to contain the true intent at a user-defined confidence level ($1-\alpha$).
The CICC Workflow
- Uncertainty Mapping: The system takes heuristic scores (e.g., softmax outputs) from an existing intent classifier like BERT.
- Calibration: Using a small calibration dataset, CICC calculates a threshold based on the desired error rate ($\alpha$). For instance, setting $\alpha = 0.01$ targets a 99% coverage guarantee.
- Set Generation: For new queries, CICC identifies all intents meeting the statistical criteria for the prediction set.
- Decision Logic:
- If the set size is 1, the system proceeds with that intent.
- If the set size is between 1 and a predefined threshold ($th$), it generates a targeted clarification question.
- If the set size exceeds the threshold, the query is deemed too ambiguous or Out-of-Scope (OOS).
- Natural Language Generation: A generative language model (LLM) converts the prediction set into a natural-sounding question, such as: "I see you're having issues with a transfer. Was it a failed transfer or a declined one?"
Experimental Evaluation
The framework was rigorously tested across seven diverse datasets, including public benchmarks like ACID (insurance), ATIS (airline travel), and HWU64 (home automation), as well as IND, a real-world Dutch banking dataset from ING Group. Specialized variants (B77-OOS and C150-OOS) were utilized to evaluate the framework's performance on out-of-scope detection.
Key Results and Insights
CICC demonstrates several advantages over traditional heuristic methods. Most notably, it is one of the few frameworks that consistently meets the $1-\alpha$ coverage requirement across all tested datasets. As shown in the performance plots, the coverage (bottom row) closely tracks the target diagonal, proving the reliability of the statistical guarantees.
Efficiency and Precision
CICC generates small, precise clarification questions. In the ACID dataset, it achieved the target coverage with an average CQ size of approximately three intents. Furthermore, CICC correctly identifies a single intent more frequently than baselines, reducing the cognitive load on users by avoiding unnecessary questions.
Robust Out-of-Scope Detection
By treating extreme ambiguity as a proxy for OOS, CICC provides a robust mechanism for rejecting queries the system cannot handle. A specialized version, CICC-OOS, achieved an F1-score of 0.91 on the C150-OOS task, significantly outperforming previous state-of-the-art baselines.
Conclusion and Future Work
CICC offers a model-agnostic, statistically rigorous approach to intent classification that improves user experience by providing clear, reliable clarifications. While the framework currently focuses on single-turn clarifications, future research could explore multi-turn stopping rules and further evaluate the safety and quality of the LLM-generated natural language components. For practitioners, CICC provides a powerful tool to build more transparent and trustworthy dialogue agents without the need for expensive model retraining.
Summary updated