Sequential Short-Text Classification with Recurrent and Convolutional Neural Networks

Ji Young Lee, Franck Dernoncourt

Introduction

Short-text classification is a fundamental task in natural language processing, often applied to sentiment analysis or question answering. However, traditional approaches frequently treat utterances as isolated units, ignoring the rich contextual information inherent in human conversation. In dialog act recognition, for instance, the meaning of a simple "Yeah" depends entirely on whether it follows a question, an assertion, or a request. This paper introduces a neural framework that explicitly models the sequence of utterances to improve classification accuracy.

Utterance Representation

The first stage of the proposed model involves converting a single short-text utterance into a fixed-length vector representation ($s$). The authors evaluated two primary architectures for this "Short-text to Vector" (S2V) task: Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs), specifically LSTMs.

Utterance representation using an RNN with pooling to produce a fixed-length vector $s$.

For the RNN-based models, various pooling strategies—such as selecting the final hidden state or calculating the mean or maximum across all hidden states—were employed to summarize the sequence into a single vector. Interestingly, CNNs were found to slightly outperform LSTMs in representing short texts for this specific task.

Sequential Classification Framework

The core innovation of this work is the transition from isolated classification to sequential classification. Instead of predicting the class of an utterance based solely on its own representation, the model incorporates a "history" of preceding context. This is controlled by two hyperparameters:

  • $d_1$: The number of previous utterance representations (vectors) included.
  • $d_2$: The number of previous predicted class labels included.
Comparison of sequential classification architectures, illustrating non-sequential models versus those incorporating representation history ($d_1$) and label history ($d_2$).

These inputs are processed through a two-layer feedforward network. The authors discovered that incorporating text history ($d_1$) was generally more effective than label history ($d_2$). High-dimensional text vectors preserve more nuanced information, whereas labels are "lossy" and can lead to error propagation if a previous utterance was misclassified.

Experimental Results

The model was benchmarked against three major dialog act datasets: DSTC 4 (human-human dialogs), MRDA (multi-party meetings), and SwDA (telephone conversations). In all cases, the sequential models significantly outperformed non-sequential baselines.

  • MRDA: The CNN-based sequential model achieved 84.6% accuracy, surpassing the previous graphical model baseline of 81.3%.
  • SwDA: Reached 73.1% accuracy, a notable improvement over standard Hidden Markov Models (71.0%).
  • DSTC 4: The LSTM-based model achieved 66.2% accuracy, representing a substantial leap from the SVM baseline of 57.0%.

Conclusion

By bridging the gap between deep learning and sequential modeling, this framework demonstrates that context is vital for short-text classification in dialog. While the model achieved state-of-the-art results, future work may explore more dynamic ways to model conversation flow beyond fixed history windows to further mitigate error propagation and handle longer-range dependencies.

Summary updated

Abstract

Recent approaches based on artificial neural networks (ANNs) have shown promising results for short-text classification. However, many short texts occur in sequences (e.g., sentences in a document or utterances in a dialog), and most existing ANN-based systems do not leverage the preceding short texts when classifying a subsequent one. In this work, we present a model based on recurrent neural networks and convolutional neural networks that incorporates the preceding short texts. Our model achieves state-of-the-art results on three different datasets for dialog act prediction.
BibTeX
@inproceedings{lee2016sequential,
  author = {Ji Young Lee and Franck Dernoncourt},
  title = {{Sequential Short-Text Classification with Recurrent and Convolutional Neural Networks}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2016},
  doi = {10.18653/v1/n16-1062},
}
Figure