Conformal Intent Classification and Clarification for Fast and Accurate Intent Recognition

Floris den Hengst, Ralf Wolter, Patrick Altmeyer, Arda Kaygan

Abstract

We present Conformal Intent Classification and Clarification (CICC), a framework for fast and accurate intent classification for task-oriented dialogue systems. The framework turns heuristic uncertainty scores of any intent classifier into a clarification question that is guaranteed to contain the true intent at a pre-defined confidence level.By disambiguating between a small number of likely intents, the user query can be resolved quickly and accurately. Additionally, we propose to augment the framework for out-of-scope detection.In a comparative evaluation using seven intent recognition datasets we find that CICC generates small clarification questions and is capable of out-of-scope detection.CICC can help practitioners and researchers substantially in improving the user experience of dialogue agents with specific clarification questions.
BibTeX
@inproceedings{hengst2024conformal,
  author = {Floris den Hengst and Ralf Wolter and Patrick Altmeyer and Arda Kaygan},
  title = {{Conformal Intent Classification and Clarification for Fast and Accurate Intent Recognition}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-naacl.156},
}

Loading related content...

Figure