A New Data Augmentation Method for Intent Classification Enhancement and its Application on Spoken Conversation Datasets

Zvi Kons, Aharon Satt, Hong-Kwang Kuo, Samuel Thomas, Boaz Carmeli, Ron Hoory, Brian Kingsbury

TL;DR

Researchers at IBM have developed the Nearest Neighbors Scores Improvement (NNSI) algorithm, a weakly-supervised method that automatically labels "hard" (ambiguous) data samples to improve intent classifiers. By leveraging the consensus of semantically similar neighbors, NNSI reduces the need for manual labeling and cuts error rates in spoken conversation systems by up to 10%.

What Problem Does This Paper Address?

Building a robust intent classifier—the engine that figures out if a user wants to "check a balance" or "speak to an agent"—is a classic "cold start" problem. Before a virtual assistant launches, developers lack real-world data. Once it's live, they are flooded with unlabeled data that is often "noisy" due to Speech-to-Text (STT) errors and the informal nature of spoken language.

The standard solution is Active Learning, where a human manually labels samples the model is unsure about. However, this is incredibly time-consuming. While "self-training" (letting the model label its own high-confidence samples) exists, it offers diminishing returns because the model doesn't learn much from samples it already understands well. The real challenge is accurately labeling the high-ambiguity samples—the ones where the model is confused—without requiring a human in the loop.

Methods

The authors introduce the NNSI algorithm, which operates on the "smoothness assumption": semantically similar text samples should share the same intent.

  1. Identify Ambiguity: The system uses a baseline classifier to score unlabeled samples. It calculates an "ambiguity measure" ($\Delta$)—the difference between the highest and second-highest confidence scores. A small $\Delta$ means the model is "confused."
  2. Find Neighbors: For a high-ambiguity sample, the algorithm finds its $N$ nearest neighbors in a semantic embedding space (using a pre-trained DNN and cosine distance).
  3. Score Averaging: Instead of just looking at the labels of neighbors, NNSI averages the raw confidence score vectors of the sample and its neighbors.
  4. Thresholding: If this averaged score results in a much clearer winner (a $\Delta$ above a certain threshold), the sample is assigned that new label and added to the training set.

This differs from standard k-Nearest Neighbors because it propagates the classifier's uncertainty across the local neighborhood to reach a more stable consensus.

Datasets

The researchers evaluated NNSI on two large-scale, real-life datasets:

  • Dataset A: From a commercial voice system with 104 different intents. It included a baseline of 3,300 manually labeled samples and a pool of over 200,000 unlabeled audio conversations.
  • Dataset B: A call center dataset with 33 intents and approximately 30,000 samples.

Both datasets allowed for a comparison between "clean" manual transcriptions and "noisy" STT-generated text.

Key Insights and Ideas

  • The Value of "Hard" Samples: The paper confirms that adding low-ambiguity samples to training data provides minimal benefit. The real gains come from correctly labeling the samples the model is currently failing on.
  • Consensus as a Denoiser: NNSI acts as a filter. By averaging scores across neighbors, it can "correct" a wrong guess made on a single noisy STT string by looking at how the model reacted to similar, perhaps cleaner, phrases.
  • Bridging the Training-Deployment Gap: One of the most interesting findings is that NNSI is particularly effective at helping models trained on "clean" written text adapt to "noisy" spoken STT output. It helps "bridge the gap" by finding semantic links between the two domains.

Key Results

  • Error Reduction: Adding NNSI-labeled samples to the training set reduced error rates by up to 10% compared to the baseline.
  • Label Accuracy: For high-ambiguity samples where the baseline classifier was only 37.9% accurate, the NNSI algorithm was able to generate correct labels 67.8% of the time.
  • Scalability: The performance of the classifier improved steadily as the size of the unlabeled dataset increased, suggesting the method scales well with the "firehose" of data generated by live systems.
  • Superiority over Random: NNSI significantly outperformed both random selection and the strategy of only adding low-ambiguity (easy) samples.

Limitations and Future Work

While NNSI is effective, the authors note several areas for further exploration:

  • Graph Complexity: The current method uses a simple k-nearest neighbor approach. More complex graph-based semi-supervised learning methods might yield even better results.
  • Similarity Measures: The results are dependent on the quality of the text embeddings. Exploring different embedding models or distance metrics could refine the "neighbor" selection.
  • End-to-End Systems: The authors suggest investigating whether this logic can be applied to end-to-end intent classification (going directly from audio to intent) without the intermediate STT step.
  • Classifier Variety: The experiments focused on SVM-based classifiers; testing on modern Transformer-based architectures would be a logical next step.

Summary updated

Abstract

Intent classifiers are vital to the successful operation of virtual agent systems. This is especially so in voice activated systems where the data can be noisy with many ambiguous directions for user intents. Before operation begins, these classifiers are generally lacking in real-world training data. Active learning is a common approach used to help label large amounts of collected user input. However, this approach requires many hours of manual labeling work. We present the Nearest Neighbors Scores Improvement (NNSI) algorithm for automatic data selection and labeling. The NNSI reduces the need for manual labeling by automatically selecting highly-ambiguous samples and labeling them with high accuracy. This is done by integrating the classifier’s output from a semantically similar group of text samples. The labeled samples can then be added to the training set to improve the accuracy of the classifier. We demonstrated the use of NNSI on two large-scale, real-life voice conversation systems. Evaluation of our results showed that our method was able to select and label useful samples with high accuracy. Adding these new samples to the training data significantly improved the classifiers and reduced error rates by up to 10%.
BibTeX
@inproceedings{kons2022new,
  author = {Zvi Kons and Aharon Satt and Hong-Kwang Kuo and Samuel Thomas and Boaz Carmeli and Ron Hoory and Brian Kingsbury},
  title = {{A New Data Augmentation Method for Intent Classification Enhancement and its Application on Spoken Conversation Datasets}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=tgHrlAC7Ra0},
}

Loading related content...

Figure