Introduction
In conversational AI, "Intent Classification" is the fundamental mechanism by which chatbots interpret user requests. Traditional classifiers typically operate on a "forced choice" basis, mapping inputs to a predefined set of labels. However, this approach fails when users provide "Out-of-Scope" (OOS) inputs—queries that do not belong to any known category. Standard models often misclassify these irrelevant inputs with high confidence, leading to poor user experiences. Detecting OOS inputs is challenging because models cannot be easily trained on an exhaustive set of "everything else in the world."
Methodology: The Word Graph Space
This research proposes a novel framework that shifts from treating intent labels as discrete, independent symbols to representing them as points in a continuous Word Graph Space.
1. Graph Construction and Embedding
The authors construct a graph where both intent labels and individual words serve as nodes. An edge is established between a word and a label if that word appears in the training data for that specific intent. This structure captures the semantic "connective tissue" between different intents. Using the DeepWalk algorithm, the system performs random walks across this graph to generate vector embeddings for every node, transforming the graph into a continuous mathematical space ($\mathbb{R}^m$).
2. Regression-Based Mapping
Instead of a standard classification head, the proposed models (e.g., LSTM+ and BERT+) utilize a sentence encoder followed by a regression layer. This projects input sentences directly into the same vector space as the graph embeddings.
- In-Scope Classification: The model identifies the intent whose graph embedding has the highest dot product similarity to the sentence embedding.
- OOS Detection: If the similarity to the nearest intent falls below a defined threshold, the input is rejected as OOS.
Experimental Setup
The method was evaluated across three major benchmarks:
- CLINC150 (Larson Dataset): A standard public dataset with 150 intents and dedicated OOS queries.
- Finance Dataset: A private Portuguese dataset from a large financial institution containing 285 classes.
- Chatbot Pool: A collection of 40 diverse chatbot datasets (20 English, 20 Portuguese) used to test robustness across domains.
Key Results
The results on the Larson dataset demonstrated a significant breakthrough in OOS detection. The False Acceptance Rate (FAR)—the frequency with which OOS queries are wrongly accepted—dropped from approximately 41.1% in previous state-of-the-art models to just 9.9% with the BERT+ method. This represents a 31.2 percentage point improvement.
Furthermore, the Equal Error Rate (EER)—a balanced metric for OOS detection—improved from 12.8% for standard BERT to 9.8% for BERT+. The method's robustness was further confirmed across the 40-dataset Chatbot Pool, where the "plus" models consistently outperformed standard softmax-based classifiers.
Discussion and Limitations
A key insight of this work is that labels are defined by their constituent words. By modeling these relationships, the system learns the "semantic scope" of an intent more effectively. Notably, this improvement is achieved without requiring OOS examples during the training phase.
However, the method's efficacy is dependent on the quality of the sentence encoder. While it excels with BERT and LSTM, it showed diminished performance when paired with simpler techniques like TF-IDF, suggesting that high-quality vector representations are essential for navigating the word graph space.
Conclusion
By framing intent classification as a regression problem within a shared semantic space, this approach provides a more nuanced understanding of distance between concepts. This allows chatbots to more accurately recognize the boundaries of their knowledge, significantly reducing the error rate for out-of-scope queries across multiple languages and domains.
Summary updated