A new data augmentation method for intent classification enhancement and its application on spoken conversation datasets

Zvi Kons, Aharon Satt, Hong-Kwang Kuo, Samuel Thomas, Boaz Carmeli, Ron Hoory, Brian Kingsbury

Abstract

Intent classifiers are vital to the successful operation of virtual agent systems. This is especially so in voice activated systems where the data can be noisy with many ambiguous directions for user intents. Before operation begins, these classifiers are generally lacking in real-world training data. Active learning is a common approach used to help label large amounts of collected user input. However, this approach requires many hours of manual labeling work. We present the Nearest Neighbors Scores Improvement (NNSI) algorithm for automatic data selection and labeling. The NNSI reduces the need for manual labeling by automatically selecting highly-ambiguous samples and labeling them with high accuracy. This is done by integrating the classifier's output from a semantically similar group of text samples. The labeled samples can then be added to the training set to improve the accuracy of the classifier. We demonstrated the use of NNSI on two large-scale, real-life voice conversation systems. Evaluation of our results showed that our method was able to select and label useful samples with high accuracy. Adding these new samples to the training data significantly improved the classifiers and reduced error rates by up to 10%.

Introduction

Building robust intent classifiers—the components responsible for mapping user utterances to specific actions like "check balance" or "speak to agent"—presents a significant "cold start" challenge. Developers often lack real-world data before a system's launch. Once operational, the system is inundated with unlabeled, noisy data, frequently containing Speech-to-Text (STT) errors. While Active Learning is the standard remedy, the manual effort required for labeling is prohibitive. Traditional self-training often fails to provide significant gains because it focuses on samples the model already understands well. The core challenge lies in accurately labeling "high-ambiguity" samples—those where the model is most uncertain—without human intervention.

The NNSI Algorithm

The proposed Nearest Neighbors Scores Improvement (NNSI) algorithm leverages the "smoothness assumption": semantically similar text samples likely share the same intent. The process follows four key steps:

  1. Identify Ambiguity: A baseline classifier scores unlabeled samples. Ambiguity ($\Delta$) is defined as the difference between the top two confidence scores. A low $\Delta$ indicates high model confusion.
  2. Find Neighbors: For ambiguous samples, the algorithm identifies $N$ nearest neighbors in a semantic embedding space using a pre-trained deep neural network and cosine distance.
  3. Score Averaging: NNSI averages the raw confidence score vectors of the target sample and its neighbors. This propagates uncertainty across the local neighborhood to reach a stable consensus.
  4. Thresholding: If the averaged score yields a clear winner (a $\Delta$ above a set threshold), the sample is assigned that label and added to the training set.

Experimental Results

The researchers evaluated NNSI on two large-scale datasets: a commercial voice system with 104 intents (Dataset A) and a call center dataset with 33 intents (Dataset B). Both included manual transcriptions and noisy STT output.

Scalability and Error Reduction

A key finding was the algorithm's scalability. As the pool of unlabeled data grows, the classifier's performance improves steadily.

Error reduction relative to baseline as a function of unlabeled dataset size.

As shown in Figure 1, error reduction relative to the baseline reaches nearly 10% as the unlabeled dataset size increases. This suggests that NNSI effectively mines the "firehose" of data generated by live systems to continuously improve the model.

Performance Comparison

NNSI significantly outperformed random selection strategies and strategies focused only on "easy" (low-ambiguity) samples.

Performance comparison of NNSI against baseline and random selection strategies for STT and Transcription data.

Table 1 highlights that NNSI achieved an 8.2% error reduction on STT text and a 9.0% reduction on manual transcriptions. Notably, for high-ambiguity samples where the baseline was only 37.9% accurate, NNSI generated correct labels with 67.8% accuracy. This demonstrates the algorithm's ability to "denoise" individual samples by looking at semantic neighbors.

Conclusion

NNSI provides a powerful, weakly-supervised method for enhancing intent classifiers in noisy, real-world environments. By automatically labeling high-ambiguity samples with high precision, it bridges the gap between clean training data and noisy deployment environments. Future work will explore more complex graph-based semi-supervised learning and the application of these methods to end-to-end classification systems.

Summary updated

Abstract

Intent classifiers are vital to the successful operation of virtual agent systems. This is especially so in voice activated systems where the data can be noisy with many ambiguous directions for user intents. Before operation begins, these classifiers are generally lacking in real-world training data. Active learning is a common approach used to help label large amounts of collected user input. However, this approach requires many hours of manual labeling work. We present the Nearest Neighbors Scores Improvement (NNSI) algorithm for automatic data selection and labeling. The NNSI reduces the need for manual labeling by automatically selecting highly-ambiguous samples and labeling them with high accuracy. This is done by integrating the classifier's output from a semantically similar group of text samples. The labeled samples can then be added to the training set to improve the accuracy of the classifier. We demonstrated the use of NNSI on two large-scale, real-life voice conversation systems. Evaluation of our results showed that our method was able to select and label useful samples with high accuracy. Adding these new samples to the training data significantly improved the classifiers and reduced error rates by up to 10%.
BibTeX
@inproceedings{kons2022new,
  author = {Zvi Kons and Aharon Satt and Hong-Kwang Kuo and Samuel Thomas and Boaz Carmeli and Ron Hoory and Brian Kingsbury},
  title = {{A new data augmentation method for intent classification enhancement and its application on spoken conversation datasets}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=tgHrlAC7Ra0},
}
Figure