On the Role of Text Preprocessing in Neural Network Architectures: An Evaluation Study on Text Categorization and Sentiment Analysis

Jose Camacho-Collados, Mohammad Taher Pilehvar

TL;DR

This paper investigates the often-overlooked impact of simple text preprocessing techniques (like tokenization, lowercasing, and lemmatization) on the performance of modern neural network models for text classification and sentiment analysis. Surprisingly, a basic tokenization approach often performs as well as or better than more complex methods, especially when large datasets are available, suggesting that neural networks can handle much of the complexity themselves.

What Problem Does This Paper Address?

In Natural Language Processing (NLP), text preprocessing is a crucial initial step that transforms raw text into a format suitable for machine learning models. While traditional NLP methods relied heavily on techniques like stemming, lemmatization, and stopword removal to reduce noise and sparsity, the advent of deep learning and powerful word embeddings has shifted the focus. However, the specific impact of these preprocessing choices on modern neural architectures, particularly Convolutional Neural Networks (CNNs) and CNN-LSTMs, remains underexplored. This paper aims to fill this gap by systematically evaluating how different preprocessing strategies affect the performance of these models on standard text categorization and sentiment analysis tasks. The authors highlight that while preprocessing is important, its role might be different in the context of deep learning compared to older methods.

Methods

The authors employ two standard neural network architectures: a basic CNN model (similar to Kim, 2014) and a CNN-LSTM model that adds a recurrent layer for potentially better handling of longer sequences. Both models are trained and evaluated using various preprocessing techniques applied to the input text and, in a separate experiment, to the corpus used for training word embeddings.

The preprocessing techniques investigated are:

  • Vanilla: Simple tokenization, where text is split into words based on whitespace.
  • Lowercased: Converts all tokens to lowercase.
  • Lemmatized: Replaces words with their base or dictionary form (lemma).
  • Multiword Grouping: Combines known multiword expressions (like "United States") into a single token.

For tokenization and lemmatization, they utilize Stanford CoreNLP. For multiword grouping, they leverage phrases from pre-trained Google News Word2vec vectors.

Datasets

The study uses a combination of datasets for topic categorization and polarity detection:

Topic Categorization: * BBC News * 20News * Reuters (reduced to 8 most frequent labels) * Ohsumed (medical domain)

Polarity Detection: * RTC * IMDB * PL05 * PL04 * Stanford Phrases

These datasets represent a mix of general news, medical, and review domains, with varying sizes and evaluation splits (10-fold cross-validation or train-test splits).

Key Insights and Ideas

  • Simplicity Often Suffices: Basic tokenization (the "vanilla" setting) is surprisingly robust and often performs as well as, or even better than, more complex preprocessing techniques like lowercasing, lemmatization, or multiword grouping.
  • Data Size Matters: The impact of preprocessing is more pronounced on smaller datasets. When training data is abundant, neural networks seem to learn effectively regardless of minor preprocessing variations, suggesting their strong generalization capabilities.
  • Domain Specificity is Key: For specialized domains (like the medical dataset Ohsumed), simple tokenization can be insufficient. In such cases, techniques that reduce sparsity, like lowercasing and lemmatization, can offer significant improvements.
  • Multiword Embeddings Shine: Pre-trained word embeddings that incorporate multiword expressions (learned from a multiword-grouped corpus) perform exceptionally well even when applied to simply tokenized (vanilla) datasets. This suggests that the way embeddings are trained can compensate for the lack of explicit multiword preprocessing in the input text.
  • Consistency is Crucial: The paper emphasizes the importance of being consistent with preprocessing choices, especially when comparing different models or research findings.

Key Results

  • Overall Performance: In most datasets and tasks, the vanilla tokenization setting achieved results comparable to or better than other preprocessing methods for both CNN and CNN+LSTM models.
  • Ohsumed Anomaly: The medical dataset Ohsumed showed a significant improvement with lowercasing and lemmatization, outperforming the vanilla setting by over six points for the CNN+LSTM model.
  • Cross-Preprocessing: When evaluating embeddings trained with different preprocessing methods on vanilla tokenized datasets, multiword-enhanced embeddings consistently yielded better results across most datasets, outperforming even the same multiword-grouped preprocessed datasets.
  • Variability: The study found a notable variability in results (up to ±2.4% on average) across different preprocessing techniques, underscoring their importance.

Limitations and Future Work

The authors acknowledge that their study focuses on simple preprocessing techniques and standard word-based neural models. They suggest future work could extend this analysis to:

  • Other Tasks: Including question answering and other NLP applications.
  • Other Languages: Particularly morphologically rich languages where preprocessing might have a different impact.
  • More Techniques: Investigating stopword removal, part-of-speech tagging, and more advanced multiword extraction methods.
  • Deeper Understanding: Further experimentation to fully grasp the significance of these results and their implications for model design and evaluation.

Summary updated

Abstract

Text preprocessing is often the first step in the pipeline of a Natural Language Processing (NLP) system, with potential impact in its final performance. Despite its importance, text preprocessing has not received much attention in the deep learning literature. In this paper we investigate the impact of simple text preprocessing decisions (particularly tokenizing, lemmatizing, lowercasing and multiword grouping) on the performance of a standard neural text classifier. We perform an extensive evaluation on standard benchmarks from text categorization and sentiment analysis. While our experiments show that a simple tokenization of input text is generally adequate, they also highlight significant degrees of variability across preprocessing techniques. This reveals the importance of paying attention to this usually-overlooked step in the pipeline, particularly when comparing different models. Finally, our evaluation provides insights into the best preprocessing practices for training word embeddings.
BibTeX
@inproceedings{camachocollados2018role,
  author = {Jose Camacho-Collados and Mohammad Taher Pilehvar},
  title = {{On the Role of Text Preprocessing in Neural Network Architectures: An Evaluation Study on Text Categorization and Sentiment Analysis}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure