Abstract
Preprocessing is a standard step in Natural Language Processing (NLP), yet its impact on the initial phase of word representation learning is often overlooked. This study systematically evaluates the effect of preprocessing large-scale training corpora—rather than just downstream classification data—on affective tasks such as sentiment analysis, emotion classification, and sarcasm detection. By integrating domain-specific techniques like negation handling and part-of-speech filtering into the representation learning phase, the authors demonstrate significant performance improvements over state-of-the-art pretrained embeddings.
Introduction
Word embeddings like Word2Vec and BERT are foundational to modern NLP, relying on the distributional hypothesis where words in similar contexts share similar representations. However, this approach often fails in affective tasks. For instance, "happy" and "sad" may appear in identical syntactic structures, leading a general model to place them close together in vector space despite their polar opposite sentiments.
While researchers typically preprocess final classification datasets (e.g., removing URLs or stemming), few investigate "upstream" preprocessing of the massive corpora used to generate the embeddings themselves. This paper addresses this gap through a large-scale analysis of how upstream preprocessing affects downstream performance in emotion-heavy tasks.
Methodology: Pre vs. Post Preprocessing
The core of the study involves comparing two distinct pipelines:
- Pre (Upstream): Preprocessing the large-scale training corpus before learning word representations.
- Post (Downstream): Preprocessing only the specific classification dataset used for the final task.
The authors evaluated several factors, both individually and in combination:
- Negation Handling: Using WordNet to transform phrases like "not happy" into "sad" before training.
- Parts-of-Speech (POS) Filtering: Retaining only nouns, verbs, adjectives, and adverbs, which carry the most affective weight.
- Standard NLP Steps: Including spellchecking, stopword removal, stemming, and basic cleaning (HTML removal, lowercasing).
These techniques were applied to three major architectures: CBOW, Skip-gram, and BERT.
Experimental Setup
The study utilized two massive training corpora: a "News" corpus (142k articles) and a "Wikipedia" corpus (23 million articles). Performance was evaluated across nine datasets categorized into:
- Sentiment Analysis: IMDB (reviews), SemEval (tweets), and Airline (tweets).
- Emotion Classification: ISEAR (narratives), Alm (fairy tales), and SSEC (tweets).
- Sarcasm Detection: Onion (headlines), IAC (responses), and Reddit (comments).
Key Insights
- The Upstream Advantage: Preprocessing the training corpus ("Pre") consistently outperformed preprocessing the classification data ("Post"). Embedding linguistic knowledge during the representation learning phase is more effective than attempting to fix text at the final classification step.
- Critical Role of Negation: Negation handling was the most impactful single step. Resolving "not" into an antonym prevents the model from grouping opposite sentiments together.
- Nuance in Stemming and Stopwords: Unlike general NLP tasks, removing stopwords or stemming often yielded mixed results in affective tasks, as these steps can strip away essential emotional nuance.
Results and Performance
The custom-preprocessed models achieved significant gains over "off-the-shelf" embeddings like GloVe, FastText, and DeepMoji.
A preprocessed BERT model emerged as the top performer on 8 out of 9 datasets. For example, on the IMDB dataset, the F-score improved from a baseline of 90.11 to 94.18. Furthermore, embeddings trained on the larger Wikipedia corpus significantly outperformed those trained on the News corpus, highlighting that scale remains a critical factor even when preprocessing is optimized.
Conclusion and Future Work
This research underscores the importance of "upstream" preprocessing for affective NLP. While the current negation handling relies on dictionary-based approaches like WordNet—which may struggle with slang—the results are compelling. Future work could explore automated selection of preprocessing factors or more complex combinations to further refine word representations for nuanced human emotions.
Summary updated