Rethinking Complex Neural Network Architectures for Document Classification

Ashutosh Adhikari, Achyudh Ram, Raphael Tang, Jimmy Lin

Introduction

In the pursuit of higher accuracy, Natural Language Processing (NLP) architectures have become increasingly complex, often incorporating multi-layered hierarchies and intricate attention mechanisms. This paper addresses the phenomenon of "complexity creep," where the marginal gains of these elaborate models may not justify their computational cost or the difficulty of reproducing their results. The authors pose a fundamental question: Is this architectural complexity truly necessary for document classification, or can simpler models achieve comparable performance through better training and regularization?

Methodology: A Minimalist Architecture

The researchers propose a streamlined model to demonstrate that architectural innovation is often secondary to robust training practices. Their approach centers on a single-layer Bi-directional Long Short-Term Memory (BiLSTM) network.

The proposed single-layer BiLSTM architecture with max-pooling.

As illustrated in the architecture diagram above, the model processes input word embeddings (a) through a single BiLSTM layer (b). The resulting forward and backward hidden states (c, d) are then aggregated using a max-pooling operation (e) to generate a fixed-length document vector (f). This vector is finally passed to an output layer (g) for classification. Notably, this design completely eschews attention mechanisms and hierarchical structures.

The Role of Regularization

The "secret sauce" of the proposed model lies in three specific regularization techniques borrowed from language modeling research:

  1. Weight-dropped LSTM: Applying DropConnect to the hidden-to-hidden weight matrices to prevent over-reliance on specific recurrent connections.
  2. Embedding Dropout: Dropping entire word embeddings during training to force the model to remain robust even when key words are absent.
  3. Temporal Averaging (TA): A post-training optimization that averages model weights over the final training steps to improve generalization and reduce noise.

Experimental Results

The authors evaluated the regularized BiLSTM ($LSTM_{reg}$) against four standard benchmarks: Reuters-21578, AAPD, IMDB, and Yelp 2014. To ensure transparency and reproducibility, they released their implementation via the Hedwig toolkit.

Experimental results comparing the regularized BiLSTM against state-of-the-art models.

As shown in the results table, the $LSTM_{reg}$ model consistently outperformed or matched significantly more complex architectures. On the Reuters dataset, it achieved an $F_1$ score of 87.0, surpassing the Hierarchical Attention Network (HAN) and the Sequence Generation Model (SGM). On the IMDB sentiment analysis task, it reached 52.8% accuracy, notably higher than the original HAN results.

Interestingly, the study also revealed that "old-school" baselines like Support Vector Machines (SVM) remain surprisingly competitive, sometimes outperforming complex neural models on specific datasets like Reuters.

Key Insights and Conclusion

The findings suggest that regularization is often more impactful than architecture. The addition of proper regularization yielded an average increase of 1.5 points in $F_1$ score, proving that how a model is trained is as critical as what is built.

By demonstrating that a simple, single-layer BiLSTM can match state-of-the-art performance, this paper highlights the need for the NLP community to prioritize reproducibility and efficiency. While complex models like Transformers continue to evolve, this work serves as a reminder that well-tuned, simpler architectures remain powerful and efficient tools for document classification.

Summary updated

BibTeX
@inproceedings{adhikari2019rethinking,
  author = {Ashutosh Adhikari and Achyudh Ram and Raphael Tang and Jimmy Lin},
  title = {{Rethinking Complex Neural Network Architectures for Document Classification}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/n19-1408},
}
Figure