Abstract
Though some recent works focus on injecting sentiment knowledge into pre-trained language models, they usually design mask and reconstruction tasks in the post-training phase. In this paper, we aim to benefit from sentiment knowledge in a lighter way. To achieve this goal, we study sentence-level sentiment analysis and, correspondingly, propose two sentiment-aware auxiliary tasks named sentiment word cloze and conditional sentiment prediction. The first task learns to select the correct sentiment words within the input, given the overall sentiment polarity as prior knowledge. On the contrary, the second task predicts the overall sentiment polarity given the sentiment polarity of the word as prior knowledge. In addition, two kinds of label combination methods are investigated to unify multiple types of labels in each task. The experimental results demonstrate that our approach consistently outperforms pre-trained models and is additive to existing knowledge-enhanced post-trained models.
Introduction
Pre-trained models (PTMs) such as BERT and RoBERTa have revolutionized Natural Language Processing (NLP), yet they often lack explicit sentiment knowledge. Existing methods to address this typically involve a computationally intensive post-training phase on large domain-specific corpora using masking and reconstruction tasks. However, these approaches face several challenges: high computational overhead due to large vocabulary predictions, heavy reliance on potentially noisy lexicons, and information loss from masking the very sentiment signals (e.g., "excellent") the model needs to learn.
KESA (Knowledge Enhanced Sentiment Analysis) proposes a "lighter" alternative by injecting sentiment lexicon knowledge during the fine-tuning phase. By avoiding the [MASK] token and utilizing efficient auxiliary tasks, KESA preserves semantic integrity and achieves state-of-the-art results without requiring massive external data.
Methodology
KESA integrates two novel auxiliary tasks into the standard sentiment classification framework to leverage local and global sentiment information simultaneously, inspired by Hebbian theory where "cells that fire together, wire together."
Sentiment Word Cloze (SWC)
In the SWC task, the model is provided with a sentence and its overall sentiment polarity (e.g., "Positive"). The objective is to identify the correct sentiment word from a small set of candidates (one ground truth and $K$ distractors). This approach is significantly more efficient than predicting a word from the entire vocabulary, using only about 0.7% of the parameters required by traditional reconstruction tasks.
Conditional Sentiment Prediction (CSP)
The CSP task provides the sentiment polarity of a specific word (derived from a lexicon) as prior information at the input stage. The model then predicts the overall sentiment of the sentence. By treating lexicon data as a "hint" rather than absolute ground truth, the model becomes more robust to inaccuracies in the sentiment lexicon.
Experimental Results
The performance of KESA was evaluated on four standard benchmarks: MR, SST2, SST5, and IMDB, using SentiWordNet 3.0 as the lexicon source. KESA consistently outperformed vanilla PTMs and specialized post-trained models like SentiLARE and SentiX.
Key findings include:
- Accuracy Gains: KESA achieved absolute improvements of 0.76% on MR, 0.38% on SST2, and 0.72% on SST5 over strong baselines.
- Task Synergy and Competition: While both SWC and CSP tasks contribute to performance, they can occasionally compete for the model's attention. The balance weight ($\gamma$) in the joint loss function is critical for optimizing performance, as shown in the sensitivity analysis across different datasets.
Conclusion
KESA demonstrates that sentiment knowledge can be effectively injected into PTMs through lightweight auxiliary tasks during fine-tuning. By focusing on efficiency and preserving semantic signals, KESA offers a scalable alternative to data-hungry post-training methods. Future work will explore resolving task competition and extending this framework to aspect-level sentiment analysis.
The code and data for this project are available at https://github.com/lshowway/KESA.
Summary updated