Sequential Short-Text Classification with Recurrent and Convolutional Neural Networks

Ji Young Lee, Franck Dernoncourt

Abstract

Recent approaches based on artificial neural networks (ANNs) have shown promising results for short-text classification. However, many short texts occur in sequences (e.g., sentences in a document or utterances in a dialog), and most existing ANN-based systems do not leverage the preceding short texts when classifying a subsequent one. In this work, we present a model based on recurrent neural networks and convolutional neural networks that incorporates the preceding short texts. Our model achieves state-of-the-art results on three different datasets for dialog act prediction.

Introduction

Short-text classification is a fundamental task in Natural Language Processing (NLP), encompassing applications from sentiment analysis to intent recognition in chatbots. A significant challenge in this domain is that short texts, such as dialog utterances, often lack sufficient information when viewed in isolation. For instance, the utterance "Okay" can function as an agreement, a question, or a conversational closer depending entirely on the preceding context. While traditional methods like Hidden Markov Models (HMMs) attempted to capture these dependencies, early deep learning models frequently treated sentences as independent units. This paper addresses this gap by proposing a neural architecture that explicitly models the sequential nature of short texts.

Model Architecture

The proposed model consists of two primary components: a sentence encoder and a sequential classification layer.

Short-Text Representation (S2V)

The first stage involves converting a variable-length short text $X_i$ into a fixed-size vector representation $s_i$. The authors evaluate two distinct architectures for this "Sentence-to-Vector" (S2V) mapping:

  • Recurrent Neural Networks (RNN/LSTM): These process words sequentially to capture long-range dependencies, followed by a pooling layer to aggregate hidden states.
  • Convolutional Neural Networks (CNN): These use filters to identify local n-gram patterns, which are then aggregated via max-pooling.
The model supports two types of sentence encoders: RNN-based (left) and CNN-based (right).

Sequential Classification Layer

The core innovation is the integration of context from preceding utterances. Instead of classifying $s_i$ in isolation, the model incorporates a window of history defined by two hyperparameters:

  • $d_1$ (Representation History): The number of preceding sentence vectors ($s_{i-1}, s_{i-2}, \dots$) fed into the classifier.
  • $d_2$ (Label History): The number of preceding predicted labels ($y_{i-1}, y_{i-2}, \dots$) used as additional features.
Different model configurations ranging from isolation (left) to full sequential context using both previous representations and labels (right).

Experimental Results

The model was evaluated on three major dialog act datasets: DSTC 4, MRDA, and SwDA. The results demonstrated that incorporating context ($d_1 + d_2 > 0$) consistently yields superior performance compared to isolated classification.

Dataset Previous Best Proposed Model (SOTA)
MRDA 82.0% (Naive Bayes) 84.6%
SwDA 72.3% (Memory-based) 73.1%
DSTC 4 57.0% (SVM) 66.2%

Key Findings

  1. Contextual Importance: Adding just one or two preceding sentences significantly boosted accuracy by 1.3 to 4.2 percentage points.
  2. Representations vs. Labels: Using previous sentence vectors ($d_1$) proved more effective than using previous labels ($d_2$). The authors suggest that high-dimensional vectors preserve more nuance, whereas labels are "lossy" and prone to error propagation.
  3. CNN Superiority: Surprisingly, CNN-based encoders generally outperformed LSTMs for short-text representation, suggesting that local phrase identification is often more critical than full sequence modeling for short utterances.

Conclusion

By bridging the gap between deep learning and sequential modeling, this work establishes a new benchmark for short-text classification. The findings highlight that for dialog-based tasks, the meaning of an utterance is inextricably linked to its history. Future work may explore more complex sequential layers, such as hierarchical RNNs, to model even longer conversational dependencies.

Summary updated

Abstract

Recent approaches based on artificial neural networks (ANNs) have shown promising results for short-text classification. However, many short texts occur in sequences (e.g., sentences in a document or utterances in a dialog), and most existing ANN-based systems do not leverage the preceding short texts when classifying a subsequent one. In this work, we present a model based on recurrent neural networks and convolutional neural networks that incorporates the preceding short texts. Our model achieves state-of-the-art results on three different datasets for dialog act prediction.
BibTeX
@inproceedings{lee2016sequential,
  author = {Ji Young Lee and Franck Dernoncourt},
  title = {{Sequential Short-Text Classification with Recurrent and Convolutional Neural Networks}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2016},
  doi = {10.18653/v1/n16-1062},
}
Figure