Subword Semantic Hashing for Intent Classification on Small Datasets

Kumar Shridhar, Ayushman Dash, Amit Sahu, Gustav Grund Pihlgren, Pedro Alonso, Vinaychandran Pondenkandath, György Kovács, Foteini Simistira, Marcus Liwicki

TL;DR

This paper introduces Subword Semantic Hashing (SSH), a novel method for extracting features from text that excels at intent classification, especially for chatbots and other applications with limited training data. SSH achieves state-of-the-art performance on three benchmark datasets by effectively handling out-of-vocabulary words and spelling errors, outperforming existing methods.

What Problem Does This Paper Address?

Intent classification, a crucial task for applications like chatbots and customer support systems, typically relies on deep learning models. However, these models are notoriously data-hungry and prone to overfitting when trained on small datasets. This is a significant challenge for real-world intent classification scenarios, where obtaining large, labeled datasets can be difficult and expensive. Existing word embedding methods also struggle with out-of-vocabulary (OOV) terms and user-generated text, which often contains spelling errors, further exacerbating the problem with small datasets. The authors aim to develop a robust feature extraction method that can achieve high accuracy even with limited training data and inherent data imperfections.

Methods

The core innovation is Subword Semantic Hashing (SSH). Inspired by the Deep Semantic Similarity Model, SSH treats sub-word tokens (trigrams of characters within words, with '#' prepended and appended) as features. This approach is fundamentally different from traditional word embeddings.

Here's a breakdown of the SSH process:

  1. Subword Tokenization: Each word in a sentence is broken down into overlapping trigrams of characters. For example, "flying" becomes "#fl", "fly", "lying", "ing#". This inherently handles OOV words because even if a whole word is unknown, its constituent subwords might be recognized.
  2. Vector Space Model (VSM): These subword tokens are then used to build a VSM. This VSM acts as a hashing function, converting input text into a feature vector.
  3. TF-IDF Vectorization: To refine the VSM, the authors employ Term Frequency-Inverse Document Frequency (TF-IDF). This weighting scheme helps to balance the importance of frequently occurring, less informative subword tokens (like "#a#") with rarer, more discriminative ones.
  4. Normalization: The resulting TF-IDF vectors are normalized using the Euclidean norm to produce the final feature vectors.

These SSH features are then fed into various standard classifiers (Ridge Classifier, SVM, MLP, etc.) for intent classification.

Datasets

The authors evaluated their method on three publicly available datasets:

  • Chatbot Corpus: Questions for a Telegram chatbot designed to answer queries about Munich's public transport. It has two intents: "Departure Time" and "Find Connection."
  • AskUbuntu Corpus: Questions extracted from the AskUbuntu platform, with five intents including "Make Update," "Setup Printer," and "Software Recommendation."
  • Web Applications Corpus: A corpus with eight intents, such as "Change Password," "Delete Account," and "Download Video."

All datasets were preprocessed by converting text to lowercase, replacing pronouns with '-PRON-', and removing special characters. To address class imbalance in smaller datasets, they employed data augmentation using dictionary-based synonym replacement (WordNet) and oversampling.

Key Insights and Ideas

  • Subword Hashing for Robustness: The key insight is that by focusing on subword units (character trigrams), the model becomes inherently more robust to spelling errors and out-of-vocabulary words. This is a significant advantage over word-level embeddings, especially for user-generated content.
  • Small Data Advantage: SSH's ability to learn meaningful features from subword units makes it particularly well-suited for small datasets, where traditional deep learning models often fail due to overfitting.
  • Versatility with Classifiers: The SSH features proved effective with a variety of standard classifiers, demonstrating the method's flexibility and the quality of the learned representations. The authors found that even simpler classifiers like Ridge Classifier and Passive Aggressive Classifier achieved excellent results when paired with SSH features.
  • Speed and Efficiency: The method is not only accurate but also fast, with inference times in the order of milliseconds, making it suitable for real-time applications.

Key Results

The paper reports state-of-the-art performance on all three benchmark datasets.

  • The authors achieved a micro F1-score of 0.996 on the Chatbot dataset, 0.94 on AskUbuntu, and 0.85 on Web Applications.
  • Their overall average performance across all datasets was 0.93, and their best performance was 0.94.
  • These results outperformed existing NLU services and open-source platforms like Botfuel, Dialogflow, Luis, Watson, Rasa, and Snips, as well as a recently published TildeCNN classifier.
  • The method demonstrated extremely low variance (< 10⁻³) between experimental runs, indicating high stability.
  • Inference times were consistently in the milliseconds range, highlighting the method's efficiency.

Limitations and Future Work

  • Domain Independence: While the method performed well across three diverse datasets, the authors suggest further testing and benchmarking on more datasets to assess its domain independence.
  • Comparison with Other Feature Extraction Methods: A future direction is to compare SSH directly with other advanced feature extraction techniques to determine scenarios where SSH is the preferred choice.
  • Integration into Conversational Services: The authors envision integrating this solution into real-time conversational services.

Summary updated

Abstract

In this paper, we introduce the use of Semantic Hashing as embedding for the task of Intent Classification and achieve state-of-the-art performance on three frequently used benchmarks. Intent Classification on a small dataset is a challenging task for data-hungry state-of-the-art Deep Learning based systems. Semantic Hashing is an attempt to overcome such a challenge and learn robust text classification. Current word embedding based are dependent on vocabularies. One of the major drawbacks of such methods is out-of-vocabulary terms, especially when having small training datasets and using a wider vocabulary. This is the case in Intent Classification for chatbots, where typically small datasets are extracted from internet communication. Two problems arise by the use of internet communication. First, such datasets miss a lot of terms in the vocabulary to use word embeddings efficiently. Second, users frequently make spelling errors. Typically, the models for intent classification are not trained with spelling errors and it is difficult to think about ways in which users will make mistakes. Models depending on a word vocabulary will always face such issues. An ideal classifier should handle spelling errors inherently. With Semantic Hashing, we overcome these challenges and achieve state-of-the-art results on three datasets: AskUbuntu, Chatbot, and Web Application. Our benchmarks are available online: https://github.com/kumar-shridhar/Know-Your-Intent
BibTeX
@inproceedings{shridhar2019subword,
  author = {Kumar Shridhar and Ayushman Dash and Amit Sahu and Gustav Grund Pihlgren and Pedro Alonso and Vinaychandran Pondenkandath and György Kovács and Foteini Simistira and Marcus Liwicki},
  title = {{Subword Semantic Hashing for Intent Classification on Small Datasets}},
  booktitle = {IJCNN},
  year = {2019},
  doi = {10.1109/ijcnn.2019.8852420},
}
Figure