On the Role of Text Preprocessing in Neural Network Architectures: An Evaluation Study on Text Categorization and Sentiment Analysis

Jose Camacho-Collados, Mohammad Taher Pilehvar

This paper investigates a fundamental yet often overlooked question in modern Natural Language Processing (NLP): does the way we clean and format text—our "preprocessing"—actually matter for deep learning models? While traditional linear models (like SVMs) rely heavily on techniques like lemmatization to handle sparse data, the deep learning community has largely assumed that neural networks and word embeddings can handle raw text just fine.

Camacho-Collados and Pilehvar put this assumption to the test, evaluating how simple decisions like lowercasing or grouping multiword expressions affect the performance of Convolutional Neural Networks (CNNs) across various text classification tasks.

TL;DR

While "vanilla" tokenization is generally sufficient for large datasets, preprocessing decisions can cause performance swings of over 2% on smaller benchmarks. Surprisingly, training word embeddings on multiword-grouped text improves performance even when the downstream task uses simple tokenization, offering a potential explanation for the long-standing success of Google’s Word2vec vectors.

What Problem Does This Paper Address?

In the "pre-deep learning" era, text preprocessing was a science in itself. To combat data sparsity, researchers would lowercase everything, remove suffixes (stemming), or convert words to their base forms (lemmatization).

With the advent of neural networks and dense word embeddings, the narrative shifted. The prevailing wisdom suggested that these models are powerful enough to learn that "Apple," "apple," and "apples" are related without manual intervention. Consequently, preprocessing has become an afterthought, often poorly documented in research papers. This paper addresses this lack of empirical evidence, asking if these "small" decisions are actually introducing hidden variables that affect model comparisons and final accuracy.

Methods

The authors evaluated four common preprocessing techniques:

  1. Vanilla: Simple tokenization (splitting by whitespace/punctuation).
  2. Lowercasing: Converting all text to lowercase.
  3. Lemmatizing: Reducing words to their dictionary form (e.g., "asking" becomes "ask").
  4. Multiword Grouping: Identifying common phrases (like "United States") and treating them as a single token ("United_States").

They tested these on two standard neural architectures: a CNN and a CNN+LSTM hybrid. The study explored two main scenarios:

  • Consistent Preprocessing: The corpus used to train word embeddings and the task dataset were preprocessed the same way.
  • Cross-Preprocessing: The task dataset remained "vanilla," but the word embeddings were trained on differently preprocessed corpora.

Datasets

The researchers used a diverse set of nine benchmarks covering two primary tasks:

  • Topic Categorization: BBC News, 20News, Reuters, and Ohsumed (a specialized medical dataset).
  • Polarity Detection (Sentiment Analysis): RTC (Rotten Tomatoes), IMDB, PL05, PL04, and Stanford Phrases (SF).
Overview of the evaluation datasets used in the study.

These datasets vary significantly in size, ranging from just 2,000 documents (PL04) to 438,000 snippets (RTC), allowing the authors to observe how data scale influences the impact of preprocessing.

Key Insights and Ideas

  • The "Vanilla" Baseline is Robust but Variable: For most general-domain tasks, simple tokenization is a strong choice. However, the authors found an average variability of ±2.4% in accuracy depending on the preprocessing method used. This suggests that "state-of-the-art" gains reported in some papers might actually be due to preprocessing choices rather than architectural innovations.
  • Data Scale Matters: Preprocessing is much more critical when training data is limited. In large datasets like RTC or Stanford Phrases, the models are robust enough to overcome the noise of different word forms. In smaller datasets, a poor preprocessing choice can significantly hurt performance.
  • Domain Specificity Changes the Rules: The medical dataset (Ohsumed) was a major outlier. Here, vanilla tokenization failed significantly. Because the word embeddings were trained on general web text, they couldn't bridge the gap to specialized medical terms. In this case, sparsity-reducing techniques like lowercasing and lemmatization provided a massive boost (over 6 percentage points).
  • The Multiword Advantage: One of the most interesting findings is that training embeddings with multiword grouping helps even if the downstream task doesn't use it. By treating "Big Apple" as one unit during embedding training, the model learns a "cleaner" representation for the word "apple" in isolation, free from the "noise" of the phrase.

Key Results

The experimental results highlight that while there is no "one-size-fits-all" strategy, the impact of these decisions is non-negligible.

Accuracy results for consistent preprocessing across CNN and CNN+LSTM models.

As seen in Table 2, the CNN+LSTM model shows significant sensitivity. In the Ohsumed (medical) dataset, the jump from Vanilla (30.8%) to Lowercased (37.5%) is dramatic.

Results for cross-preprocessing, where embeddings are preprocessed differently than the task data.

Table 3 shows the "Cross-preprocessing" results. Note how the "Multiword" row for the CNN model achieves the best overall performance in seven out of nine datasets, even though the task data itself was only vanilla-tokenized.

Limitations and Future Work

The authors acknowledge several boundaries to their study:

  • Linguistic Scope: The study focused exclusively on English. The impact of lemmatization is likely much higher in morphologically rich languages (like Turkish or Finnish).
  • Encoding Levels: The work focused on word-based models. It does not account for character-level or subword-based models (like BPE used in modern Transformers), which handle sparsity differently.
  • Task Variety: The evaluation was limited to classification. Future work could explore sequence labeling (like NER) or generative tasks.

Ultimately, this paper serves as a call to action for NLP researchers: report your preprocessing. What seems like a trivial cleanup step might be doing more heavy lifting than you think.

Summary updated

Abstract

Text preprocessing is often the first step in the pipeline of a Natural Language Processing (NLP) system, with potential impact in its final performance. Despite its importance, text preprocessing has not received much attention in the deep learning literature. In this paper we investigate the impact of simple text preprocessing decisions (particularly tokenizing, lemmatizing, lowercasing and multiword grouping) on the performance of a standard neural text classifier. We perform an extensive evaluation on standard benchmarks from text categorization and sentiment analysis. While our experiments show that a simple tokenization of input text is generally adequate, they also highlight significant degrees of variability across preprocessing techniques. This reveals the importance of paying attention to this usually-overlooked step in the pipeline, particularly when comparing different models. Finally, our evaluation provides insights into the best preprocessing practices for training word embeddings.
BibTeX
@inproceedings{camachocollados2018role,
  author = {Jose Camacho-Collados and Mohammad Taher Pilehvar},
  title = {{On the Role of Text Preprocessing in Neural Network Architectures: An Evaluation Study on Text Categorization and Sentiment Analysis}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}

Loading related content...

Figure