Revisiting LSTM Networks for Semi-Supervised Text Classification via Mixed Objective Function

Devendra Singh Sachan, Manzil Zaheer, Ruslan Salakhutdinov

Abstract

In this paper, we study bidirectional LSTM network for the task of text classification using both supervised and semisupervised approaches. Several prior works have suggested that either complex pretraining schemes using unsupervised methods such as language modeling or complicated models are necessary to achieve a high classification accuracy. However, we develop a training strategy that allows even a simple BiLSTM model, when trained with cross-entropy loss, to achieve competitive results compared with more complex approaches. Furthermore, in addition to cross-entropy loss, by using a combination of entropy minimization, adversarial, and virtual adversarial losses for both labeled and unlabeled data, we report state-of-the-art results for text classification task on several benchmark datasets. In particular, on the ACL-IMDB sentiment analysis and AG-News topic classification datasets, our method outperforms current approaches by a substantial margin. We also show the generality of the mixed objective function by improving the performance on relation extraction task.

Introduction

In the quest for higher accuracy in Natural Language Processing (NLP), the field has trended toward increasingly complex solutions: deep convolutional networks, multi-layer LSTMs, or computationally expensive unsupervised pre-training. These approaches are often slow to train, memory-intensive, and difficult to optimize. This research challenges the assumption that complexity is a prerequisite for performance, investigating whether a basic bidirectional LSTM (BiLSTM) can become competitive through improved training strategies and the effective utilization of unlabeled data.

Methodology

The core model is a standard one-layer BiLSTM with max-pooling. The innovation lies in the training strategy and the introduction of a Mixed Objective Function ($L_{MIXED}$).

Architecture of the simple BiLSTM model with max-pooling over time.

Training Strategy

Unlike many implementations that truncate long sequences to save memory, the authors backpropagate through the entire text sequence. To manage memory, they employ dynamic batching, varying the number of documents in a mini-batch to maintain a fixed total number of tokens. The strategy also emphasizes fine-tuning word embeddings and utilizing a large vocabulary that includes rare words rather than just high-frequency ones.

Mixed Objective Function

The model minimizes a weighted sum of four distinct loss functions:

  • Maximum Likelihood ($L_{ML}$): Standard cross-entropy on labeled data.
  • Adversarial Training ($L_{AT}$): Adds calculated perturbations to word embeddings to increase robustness against noisy inputs.
  • Virtual Adversarial Training ($L_{VAT}$): Applied to unlabeled data, encouraging consistent outputs for perturbed inputs to smooth the loss surface.
  • Entropy Minimization ($L_{EM}$): Encourages confident predictions on unlabeled data, effectively acting as a form of soft clustering.

Experimental Results

The approach was evaluated on seven benchmarks, including sentiment analysis (ACL-IMDB, Elec) and topic classification (AG-News, DBpedia).

Test error comparison across different training objectives over 50 epochs.

Key Insights

  • Regularization via Unsupervised Loss: The combination of $L_{VAT}$ and $L_{EM}$ acts as a powerful regularizer. While standard supervised training ($L_{ML}$) begins to overfit after 5 epochs, the mixed objective allows continued generalization for up to 50 epochs, as shown in Figure 2.
  • Scaling Performance: In contrast to purely supervised settings where larger models often overfit, the mixed objective allows performance to improve as hidden layer size or depth (up to 3 layers) increases.
  • State-of-the-Art Results: The model achieved a 4.32% error rate on ACL-IMDB, a 26.9% relative reduction in error compared to previous complex models, outperforming approaches like ULMFit.
  • Generality: The mixed objective also improved performance on relation extraction tasks (TACRED and SemEval-2010 Task 8).

Discussion and Conclusion

While the architecture is simple, the mixed objective increases computational cost; an epoch of $L_{MIXED}$ takes approximately 20 minutes compared to 3 minutes for $L_{ML}$. Additionally, performance is sensitive to the ratio of unlabeled data, peaking when labeled and unlabeled sets are roughly equal. While the model scales well to three layers, four-layer architectures exhibited instability, likely due to the sensitivity of the Entropy Minimization term. Despite these limitations, the results demonstrate that a well-trained simple model can surpass the performance of significantly more complex architectures.

Summary updated

Abstract

In this paper, we study bidirectional LSTM network for the task of text classification using both supervised and semisupervised approaches. Several prior works have suggested that either complex pretraining schemes using unsupervised methods such as language modeling (Dai and Le 2015; Miyato, Dai, and Goodfellow 2016) or complicated models (Johnson and Zhang 2017) are necessary to achieve a high classification accuracy. However, we develop a training strategy that allows even a simple BiLSTM model, when trained with cross-entropy loss, to achieve competitive results compared with more complex approaches. Furthermore, in addition to cross-entropy loss, by using a combination of entropy minimization, adversarial, and virtual adversarial losses for both labeled and unlabeled data, we report state-of-theart results for text classification task on several benchmark datasets. In particular, on the ACL-IMDB sentiment analysis and AG-News topic classification datasets, our method outperforms current approaches by a substantial margin. We also show the generality of the mixed objective function by improving the performance on relation extraction task.1
BibTeX
@inproceedings{sachan2019revisiting,
  author = {Devendra Singh Sachan and Manzil Zaheer and Ruslan Salakhutdinov},
  title = {{Revisiting LSTM Networks for Semi-Supervised Text Classification via Mixed Objective Function}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  url = {https://openreview.net/forum?id=7f5lydH-nUN},
}
Figure