Abstract
Recent approaches based on artificial neural networks (ANNs) have shown promising results for short-text classification. However, many short texts occur in sequences (e.g., sentences in a document or utterances in a dialog), and most existing ANN-based systems do not leverage the preceding short texts when classifying a subsequent one. In this work, we present a model based on recurrent neural networks and convolutional neural networks that incorporates the preceding short texts. Our model achieves state-of-the-art results on three different datasets for dialog act prediction.
Introduction
Short-text classification is a fundamental task in Natural Language Processing (NLP), encompassing applications from sentiment analysis to intent recognition in chatbots. A significant challenge in this domain is that short texts, such as dialog utterances, often lack sufficient information when viewed in isolation. For instance, the utterance "Okay" can function as an agreement, a question, or a conversational closer depending entirely on the preceding context. While traditional methods like Hidden Markov Models (HMMs) attempted to capture these dependencies, early deep learning models frequently treated sentences as independent units. This paper addresses this gap by proposing a neural architecture that explicitly models the sequential nature of short texts.
Model Architecture
The proposed model consists of two primary components: a sentence encoder and a sequential classification layer.
Short-Text Representation (S2V)
The first stage involves converting a variable-length short text $X_i$ into a fixed-size vector representation $s_i$. The authors evaluate two distinct architectures for this "Sentence-to-Vector" (S2V) mapping:
- Recurrent Neural Networks (RNN/LSTM): These process words sequentially to capture long-range dependencies, followed by a pooling layer to aggregate hidden states.
- Convolutional Neural Networks (CNN): These use filters to identify local n-gram patterns, which are then aggregated via max-pooling.
Sequential Classification Layer
The core innovation is the integration of context from preceding utterances. Instead of classifying $s_i$ in isolation, the model incorporates a window of history defined by two hyperparameters:
- $d_1$ (Representation History): The number of preceding sentence vectors ($s_{i-1}, s_{i-2}, \dots$) fed into the classifier.
- $d_2$ (Label History): The number of preceding predicted labels ($y_{i-1}, y_{i-2}, \dots$) used as additional features.
Experimental Results
The model was evaluated on three major dialog act datasets: DSTC 4, MRDA, and SwDA. The results demonstrated that incorporating context ($d_1 + d_2 > 0$) consistently yields superior performance compared to isolated classification.
| Dataset | Previous Best | Proposed Model (SOTA) |
|---|---|---|
| MRDA | 82.0% (Naive Bayes) | 84.6% |
| SwDA | 72.3% (Memory-based) | 73.1% |
| DSTC 4 | 57.0% (SVM) | 66.2% |
Key Findings
- Contextual Importance: Adding just one or two preceding sentences significantly boosted accuracy by 1.3 to 4.2 percentage points.
- Representations vs. Labels: Using previous sentence vectors ($d_1$) proved more effective than using previous labels ($d_2$). The authors suggest that high-dimensional vectors preserve more nuance, whereas labels are "lossy" and prone to error propagation.
- CNN Superiority: Surprisingly, CNN-based encoders generally outperformed LSTMs for short-text representation, suggesting that local phrase identification is often more critical than full sequence modeling for short utterances.
Conclusion
By bridging the gap between deep learning and sequential modeling, this work establishes a new benchmark for short-text classification. The findings highlight that for dialog-based tasks, the meaning of an utterance is inextricably linked to its history. Future work may explore more complex sequential layers, such as hierarchical RNNs, to model even longer conversational dependencies.
Summary updated