Fortunately, Discourse Markers Can Enhance Language Models for Sentiment Analysis

Liat Ein-Dor, Ilya Shnayderman, Artem Spector, Lena Dankin, Ranit Aharonov, Noam Slonim

Introduction

Sentiment analysis remains a cornerstone of Natural Language Processing (NLP), yet its effectiveness is often bottlenecked by the scarcity of high-quality labeled data. While large pretrained language models (LLMs) like BERT provide a robust foundation, they typically require extensive fine-tuning on human-annotated datasets to excel in specific domains. This paper, presented at AAAI-22 by researchers from IBM, introduces a novel "inter-training" approach that leverages discourse markers (DMs) to generate massive, weakly-labeled datasets, significantly improving performance in zero-shot and few-shot scenarios.

Methodology: Inter-training with Discourse Markers

The researchers propose a three-step training pipeline designed to bridge the gap between general pretraining and task-specific sentiment analysis.

The SenDM training pipeline, illustrating the inter-training step where discourse markers (DMs) from a large corpus are used to create weakly-labeled data for sentiment model development.
  1. General Pretraining: The process begins with a standard transformer-based model, such as BERT.
  2. Inter-training: This is the core innovation. The authors identified 11 "seed" discourse markers with inherent sentiment (e.g., "fortunately" for positive, "sadly" for negative). By scanning a corpus of 400 million articles, they extracted sentences beginning with these markers. The markers were then removed, and the remaining text was used as a training example, with the marker serving as a "natural" weak label. This process yielded approximately 1.8 million "silver-standard" labeled sentences.
  3. Fine-tuning: The resulting model, SenDM, can be further refined using a minimal amount of human-labeled data for specific target tasks.

Domain Adaptation and Nuance

Beyond general markers, the authors developed a method for identifying domain-specific sentiment signals. By using the general SenDM model to "scout" for phrases that consistently precede high-confidence sentiment predictions in a new corpus (e.g., financial news), they can automatically construct specialized training sets.

This approach reveals how sentiment signals change across fields. For example, in scientific literature, the word "women" was identified as a negative marker—not due to inherent sentiment, but because papers starting with "Women..." frequently address social disparities or health crises. SenDM's ability to learn these domain-specific nuances through its scouting mechanism is a significant advantage over static sentiment lexicons.

Results and Analysis

The SenDM models were evaluated across diverse datasets, including Amazon and Yelp reviews, SST-2 movie reviews, and the Financial PhraseBank (fpb75).

Zero-Shot and Few-Shot Performance

In zero-shot evaluations, SenDM consistently outperformed vanilla BERT. The advantages were even more pronounced in few-shot settings. For instance, on the challenging financial dataset (fpb75), standard models often fail to converge with very few examples.

Few-shot accuracy on the fpb75 dataset. SenDM-base demonstrates superior performance and stability with as few as 16 examples, outperforming both vanilla BERT and domain-specific models like FinBERT.

As shown in the figure above, SenDM-base achieves high accuracy with minimal labeled data, surpassing specialized models like FinBERT, which is pretrained on financial text but lacks the task-specific inter-training provided by discourse markers. The authors noted that even SenDM-tiny (a very small version of BERT) often outperformed the standard, much larger BERT-base in these low-data regimes.

Conclusion

This research demonstrates that discourse markers are a powerful, "natural" resource for training sentiment analysis models. By treating these linguistic cues as inherent labels, researchers can build robust models that generalize well across domains and perform exceptionally in data-constrained environments. While currently focused on English, the universality of discourse markers suggests this method is a promising candidate for low-resource language adaptation and targeted sentiment analysis in future work.

Summary updated

Abstract

In recent years, pretrained language models have revolutionized the NLP world, while achieving state of the art performance in various downstream tasks. However, in many cases, these models do not perform well when labeled data is scarce and the model is expected to perform in the zero or few shot setting. Recently, several works have shown that continual pretraining or performing a second phase of pretraining (inter-training) which is better aligned with the downstream task, can lead to improved results, especially in the scarce data setting. Here, we propose to leverage sentiment-carrying discourse markers to generate large-scale weakly-labeled data, which in turn can be used to adapt language models for sentiment analysis. Extensive experimental results show the value of our approach on various benchmark datasets, including the finance domain. Code, models and data are available at https://github.com/ibm/tslm-discourse-markers.
BibTeX
@inproceedings{eindor2022fortunately,
  author = {Liat Ein-Dor and Ilya Shnayderman and Artem Spector and Lena Dankin and Ranit Aharonov and Noam Slonim},
  title = {{Fortunately, Discourse Markers Can Enhance Language Models for Sentiment Analysis}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i10.21305},
}
Figure