Theoretical and Empirical Advantages of Dense-Vector to One-Hot Encoding of Intent Classes in Open-World Scenarios

Paulo Cavalin, Claudio Santos PinhanezCOLING 2024

TL;DR

This paper argues that the success of dense-vector encodings for out-of-scope (OOS) intent detection, compared to traditional one-hot encodings, stems from their ability to create richer, more complex topological spaces for class representations. Empirically, randomly generated dense-vector encodings significantly outperform one-hot encodings in OOS detection tasks.

What Problem Does This Paper Address?

In many real-world applications, like chatbots or virtual assistants, it's crucial not only to classify user intents into known categories but also to identify when a user's request is outside the system's capabilities (Out-of-Scope or OOS). Traditional intent classification often uses one-hot encoding, where each intent is a distinct point in a high-dimensional space. While effective for in-scope (IS) classification, this method struggles with OOS detection because it assumes all known classes are equidistant and equally distinct. Recent work has shown dense-vector encodings can improve OOS detection, but the authors here propose that the benefit comes from the richer representational power of dense vectors, not necessarily from domain-specific knowledge. They aim to theoretically and empirically demonstrate this advantage.

Methods

The core of the paper lies in analyzing the "topologies" of class representations. They compare:

  • One-Hot Encoding:
    • Cmax: Using the maximum coordinate as the decision boundary.
  • Csoftmax: Using the softmax function to normalize probabilities.
  • Ca: Using Euclidean distance to one-hot vectors.
  • Dense-Vector Encoding:
    • Cr: Using Euclidean distance to dense-vector representations of classes.
  • CP: Generalizing Cr to arbitrary dimensions.

The authors theoretically analyze how these encoding methods partition the output space, particularly in the presence of an OOS class (when a threshold θ > 0). They demonstrate that one-hot methods create a limited number of distinct topologies, while dense-vector methods can generate a significantly larger, quadratically increasing number of topologies with the number of classes. Empirically, they use the Universal Sentence Encoder (USE) as the base classifier and compare the performance of one-hot softmax, one-hot distance, and dense-vector encodings (both knowledge-based word graphs and randomly generated vectors).

Datasets

The paper evaluates its methods on four public datasets:

  • HINT3: Three small, unbalanced, single-domain datasets.
  • SOFTMattress (21 intents)
    • Curekart (28 intents)
    • Powerplay11 (59 intents)
  • CLINC150: A larger, balanced dataset with 150 intents across five domains, specifically designed for OOS evaluation.

Key Insights and Ideas

  • Topology Matters: The primary insight is that the advantage of dense-vector encodings for OOS detection is largely due to their ability to create more complex and nuanced "topologies" in the output space, allowing for better separation of in-scope and out-of-scope regions.
  • One-Hot Encoding's Limitation: One-hot encodings, by their nature, impose a very limited structure (e.g., equidistant points) on the class space, restricting the types of decision boundaries that can be formed, especially for OOS detection.
  • Dense Vectors Offer Richness: Dense vectors, even when randomly generated, can represent a vastly larger number of distinct class configurations (topologies) compared to one-hot encodings. This richness is key to better OOS performance.
  • Random is Good Enough: The experiments suggest that even simple, randomly generated dense-vector encodings can yield substantial improvements in OOS detection, questioning the absolute necessity of complex domain knowledge for achieving these gains.
  • OOS vs. IS Trade-off: Improvements in OOS detection (lower FAR) often come with a slight decrease in in-scope accuracy (higher ISER), highlighting a common trade-off in open-world classification.

Key Results

  • Theoretical:
    • Cmax (one-hot) defines only 1 topology for any number of classes c.
  • Csoftmax and Ca (one-hot) define c different topologies.
  • Cr (dense-vector) defines at least c^2 different topologies, growing quadratically with the number of classes.
  • Empirical (HINT3 datasets):
    • Randomly generated dense-vector encodings (R(N)) achieved improvements in Equal Error Rate (EER) of -23% to -28% compared to one-hot softmax.
  • These improvements were primarily driven by significant reductions in False Acceptance Rate (FAR) for OOS samples.
  • Dense-vector encodings also outperformed one-hot distance and knowledge-based word graph methods in FAR.
  • Empirical (CLINC150 dataset):
    • Random dense-vector encodings (R(20)) improved EER by -35% compared to one-hot softmax.
  • FAR saw a substantial gain of -42% with R(30).
  • However, In-Scope Error Rate (ISER) showed a slight degradation, indicating the trade-off.

Limitations and Future Work

  • Finding Optimal Topologies: The paper acknowledges that while random search shows promise, efficiently finding the optimal dense-vector topology for a given task remains an open and challenging problem. Future work should focus on developing better search strategies or loss functions for this.
  • Architecture and Hyperparameter Tuning: The evaluation used a fixed architecture and parameters. Further research could explore how tailoring these to specific problems might yield even better results.
  • Baseline Comparison: While the paper isolates class representation, other strong baselines that incorporate additional mechanisms (e.g., Mahalanobis distance) were not included, which could offer further improvements.
  • LLM Integration: The findings are highly relevant to Large Language Models (LLMs). Future work should investigate how these dense-vector encoding principles can be applied and fine-tuned within LLM architectures for downstream tasks.

Summary updated

BibTeX
@inproceedings{cavalin2024theoretical,
  author = {Paulo Cavalin and Claudio Santos Pinhanez},
  title = {{Theoretical and Empirical Advantages of Dense-Vector to One-Hot Encoding of Intent Classes in Open-World Scenarios}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2024},
}
Figure