Exploring the Contribution of Unlabeled Data in Financial Sentiment Analysis

Jimmy S. J. Ren, Wei Wang, Jiawei Wang, Stephen S. Y. Liao

Abstract

Semi-supervised learning (SSL) promises to leverage abundant unlabeled data to improve model performance, yet it often results in accuracy degradation compared to supervised baselines. This paper investigates this phenomenon in the domain of financial sentiment analysis. We demonstrate that the "performance degradation" typically associated with SSL is not an inherent flaw of the data, but rather a failure of feature selection. By systematically varying vocabulary size, we show that unlabeled data can significantly boost accuracy, provided the bias-variance tradeoff is managed through optimal feature selection.

Introduction

In specialized fields like finance, obtaining labeled data is a significant bottleneck. Determining whether a sentence in a 10-K filing is "positive" or "neutral" requires domain expertise, making crowdsourcing impractical. Semi-supervised learning, which combines a small labeled dataset with a larger unlabeled one, offers a potential solution. However, the academic community has long debated its efficacy, as many studies report that adding unlabeled data can paradoxically decrease classifier accuracy. This research seeks to reconcile these views by introducing vocabulary size as a critical dimension in SSL evaluation.

Methodology

The study employs a Multinomial Naive Bayes (MNB) classifier integrated with the Expectation-Maximization (EM) algorithm to incorporate unlabeled data. Unlike traditional evaluations that focus solely on the quantities of labeled and unlabeled samples, the authors introduce a fourth dimension: Vocabulary Size (Number of Features).

The dataset comprises sentences from the Management’s Discussion and Analysis (MD&A) sections of SEC Form 10-K filings across ten different industries. Labels were verified by a Certified Public Accountant (CPA) to ensure high quality. The experiments involved 5,245 labeled sentences and 5,342 unlabeled sentences, categorized into Positive, Negative, and Neutral classes.

Key Insights and Ideas

The core finding is that semi-supervised models are exceptionally sensitive to the number of features. While a supervised model's performance might plateau or decline slowly as features are added, an SSL model often experiences a "Performance Cliff."

Accuracy vs. Vocabulary Size with 383 labeled samples.

As seen in the figure above, with a small labeled set (383 samples), the SSL curves (green, black, red) show much higher peaks than the labeled-only baseline (blue dashed), but they also exhibit much steeper declines. This suggests that "performance degradation" reported in previous literature likely occurred because researchers used a vocabulary size that was optimal for labeled data but "broke" the model once unlabeled data was introduced.

Key Results

The experiments reveal two critical phenomena:

  1. The Shifted Peak: Adding unlabeled data moves the "sweet spot" for vocabulary size. To realize the benefits of SSL, the feature set must be re-optimized for the new data distribution.
  2. Higher Performance Ceiling: When the vocabulary size is correctly tuned, the semi-supervised model consistently outperforms the supervised baseline.
Accuracy vs. Vocabulary Size with 2593 labeled samples.

In the case of a larger labeled set (2593 samples), the SSL model (black line) achieves a significantly higher peak accuracy than the baseline. The data suggests that while unlabeled data may introduce estimation bias if model assumptions are violated, this is offset by a substantial reduction in variance when the feature set is optimized.

Conclusion and Future Work

This research clarifies that unlabeled data is a valuable asset for financial sentiment analysis, provided that feature selection is treated as a primary variable. The "degradation" of SSL is a choice of suboptimal parameters rather than a failure of the method itself. Future work should focus on developing automated feature selection frameworks that utilize the pseudo-labels generated during the EM process to find the optimal vocabulary size dynamically. Additionally, exploring whether these insights scale to document-level classification remains an open question.

Summary updated

Abstract

With the proliferation of its applications in various industries, sentiment analysis by using publicly available web data has become an active research area in text classification during these years. It is argued by researchers that semi-supervised learning is an effective approach to this problem since it is capable to mitigate the manual labeling effort which is usually expensive and time-consuming. However, there was a long-term debate on the effectiveness of unlabeled data in text classification. This was partially caused by the fact that many assumptions in theoretic analysis often do not hold in practice. We argue that this problem may be further understood by adding an additional dimension in the experiment. This allows us to address this problem in the perspective of bias and variance in a broader view. We show that the well-known performance degradation issue caused by unlabeled data can be reproduced as a subset of the whole scenario. We argue that if the bias-variance trade-off is to be better balanced by a more effective feature selection method unlabeled data is very likely to boost the classification performance. We then propose a feature selection framework in which labeled and unlabeled training samples are both considered. We discuss its potential in achieving such a balance. Besides, the application in financial sentiment analysis is chosen because it not only exemplifies an important application, the data possesses better illustrative power as well. The implications of this study in text classification and financial sentiment analysis are both discussed.
BibTeX
@inproceedings{ren2013exploring,
  author = {Jimmy S. J. Ren and Wei Wang and Jiawei Wang and Stephen S. Y. Liao},
  title = {{Exploring the Contribution of Unlabeled Data in Financial Sentiment Analysis}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2013},
  doi = {10.1609/aaai.v27i1.8459},
}
Figure