Introduction
Sentiment analysis remains a cornerstone of Natural Language Processing (NLP), yet its effectiveness is often bottlenecked by the scarcity of high-quality labeled data. While large pretrained language models (LLMs) like BERT provide a robust foundation, they typically require extensive fine-tuning on human-annotated datasets to excel in specific domains. This paper, presented at AAAI-22 by researchers from IBM, introduces a novel "inter-training" approach that leverages discourse markers (DMs) to generate massive, weakly-labeled datasets, significantly improving performance in zero-shot and few-shot scenarios.
Methodology: Inter-training with Discourse Markers
The researchers propose a three-step training pipeline designed to bridge the gap between general pretraining and task-specific sentiment analysis.
- General Pretraining: The process begins with a standard transformer-based model, such as BERT.
- Inter-training: This is the core innovation. The authors identified 11 "seed" discourse markers with inherent sentiment (e.g., "fortunately" for positive, "sadly" for negative). By scanning a corpus of 400 million articles, they extracted sentences beginning with these markers. The markers were then removed, and the remaining text was used as a training example, with the marker serving as a "natural" weak label. This process yielded approximately 1.8 million "silver-standard" labeled sentences.
- Fine-tuning: The resulting model, SenDM, can be further refined using a minimal amount of human-labeled data for specific target tasks.
Domain Adaptation and Nuance
Beyond general markers, the authors developed a method for identifying domain-specific sentiment signals. By using the general SenDM model to "scout" for phrases that consistently precede high-confidence sentiment predictions in a new corpus (e.g., financial news), they can automatically construct specialized training sets.
This approach reveals how sentiment signals change across fields. For example, in scientific literature, the word "women" was identified as a negative marker—not due to inherent sentiment, but because papers starting with "Women..." frequently address social disparities or health crises. SenDM's ability to learn these domain-specific nuances through its scouting mechanism is a significant advantage over static sentiment lexicons.
Results and Analysis
The SenDM models were evaluated across diverse datasets, including Amazon and Yelp reviews, SST-2 movie reviews, and the Financial PhraseBank (fpb75).
Zero-Shot and Few-Shot Performance
In zero-shot evaluations, SenDM consistently outperformed vanilla BERT. The advantages were even more pronounced in few-shot settings. For instance, on the challenging financial dataset (fpb75), standard models often fail to converge with very few examples.
As shown in the figure above, SenDM-base achieves high accuracy with minimal labeled data, surpassing specialized models like FinBERT, which is pretrained on financial text but lacks the task-specific inter-training provided by discourse markers. The authors noted that even SenDM-tiny (a very small version of BERT) often outperformed the standard, much larger BERT-base in these low-data regimes.
Conclusion
This research demonstrates that discourse markers are a powerful, "natural" resource for training sentiment analysis models. By treating these linguistic cues as inherent labels, researchers can build robust models that generalize well across domains and perform exceptionally in data-constrained environments. While currently focused on English, the universality of discourse markers suggests this method is a promising candidate for low-resource language adaptation and targeted sentiment analysis in future work.
Summary updated