Exploring The Contribution of Unlabeled Data in Financial Sentiment\n Analysis

Jimmy Sj. Ren, Wei Wang, Jiawei Wang, Stephen Shaoyi Liao

Abstract

With the proliferation of its applications in various industries, sentiment analysis by using publicly available web data has become an active research area in text classification during these years. It is argued by researchers that semi-supervised learning is an effective approach to this problem since it is capable to mitigate the manual labeling effort which is usually expensive and time-consuming. However, there was a long-term debate on the effectiveness of unlabeled data in text classification. This was partially caused by the fact that many assumptions in theoretic analysis often do not hold in practice. We argue that this problem may be further understood by adding an additional dimension in the experiment. This allows us to address this problem in the perspective of bias and variance in a broader view. We show that the well-known performance degradation issue caused by unlabeled data can be reproduced as a subset of the whole scenario. We argue that if the bias-variance trade-off is to be better balanced by a more effective feature selection method unlabeled data is very likely to boost the classification performance. We then propose a feature selection framework in which labeled and unlabeled training samples are both considered. We discuss its potential in achieving such a balance. Besides, the application in financial sentiment analysis is chosen because it not only exemplifies an important application, the data possesses better illustrative power as well. The implications of this study in text classification and financial sentiment analysis are both discussed.

Introduction

The application of sentiment analysis to financial data, such as corporate annual reports, has gained significant traction. However, the high cost of manual labeling—often requiring domain expertise like that of a Certified Public Accountant (CPA)—makes semi-supervised learning (SSL) an attractive alternative. Despite its potential, SSL has faced a long-standing debate due to observed "performance degradation," where adding unlabeled data reduces model accuracy. This study investigates this phenomenon by introducing vocabulary size as a critical experimental dimension, framing the problem as a challenge of balancing the bias-variance tradeoff.

Methodology

The researchers utilized a Multinomial Naive Bayes (MNB) classifier integrated with the Expectation-Maximization (EM) algorithm. The core of the experimental design involves systematically varying the vocabulary size (number of features) across different ratios of labeled and unlabeled data. This approach allows for a granular view of how feature selection impacts model performance when unlabeled data is introduced.

Data Description

The study analyzed Management’s Discussion and Analysis (MD&A) sections from SEC Form 10-K filings (2011-2012) for 10 diverse companies (e.g., Microsoft, Ford, Monsanto). The dataset comprised 5,245 labeled sentences from 2012 and 5,342 unlabeled sentences from 2011. Labels (Positive, Negative, Neutral) were double-checked by a professional CPA to ensure high-quality ground truth.

Results and Discussion

The experiments reveal that performance degradation is not an inherent property of unlabeled data but a consequence of suboptimal feature selection relative to the total data volume.

Classification accuracy vs. vocabulary size across different labeled data sources (Experiment One).

As shown in the figure above, there is consistently a "sweet spot"—a specific range of vocabulary sizes—where the inclusion of unlabeled data significantly boosts classification accuracy compared to labeled data alone. Conversely, outside this range, the same unlabeled data can lead to performance degradation. SSL models exhibit higher sensitivity to the bias-variance tradeoff than supervised models; a poorly chosen vocabulary size leads to more radical accuracy drops in the SSL context than in purely supervised models. This sensitivity underscores the importance of re-balancing features when transitioning from supervised to semi-supervised learning.

Conclusion

By re-balancing the bias-variance tradeoff through effective feature selection, unlabeled data can be reliably leveraged to enhance financial sentiment analysis. The study demonstrates that the "mystery" of performance degradation can be resolved by considering the interaction between data volume and vocabulary size. This finding suggests that feature selection frameworks should incorporate information from both labeled and unlabeled sets to achieve optimal performance, turning unlabeled data from a potential foe into a valuable asset.

Summary updated

Abstract

With the proliferation of its applications in various industries, sentiment\nanalysis by using publicly available web data has become an active research\narea in text classification during these years. It is argued by researchers\nthat semi-supervised learning is an effective approach to this problem since it\nis capable to mitigate the manual labeling effort which is usually expensive\nand time-consuming. However, there was a long-term debate on the effectiveness\nof unlabeled data in text classification. This was partially caused by the fact\nthat many assumptions in theoretic analysis often do not hold in practice. We\nargue that this problem may be further understood by adding an additional\ndimension in the experiment. This allows us to address this problem in the\nperspective of bias and variance in a broader view. We show that the well-known\nperformance degradation issue caused by unlabeled data can be reproduced as a\nsubset of the whole scenario. We argue that if the bias-variance trade-off is\nto be better balanced by a more effective feature selection method unlabeled\ndata is very likely to boost the classification performance. We then propose a\nfeature selection framework in which labeled and unlabeled training samples are\nboth considered. We discuss its potential in achieving such a balance. Besides,\nthe application in financial sentiment analysis is chosen because it not only\nexemplifies an important application, the data possesses better illustrative\npower as well. The implications of this study in text classification and\nfinancial sentiment analysis are both discussed.\n
BibTeX
@inproceedings{ren2013exploring,
  author = {Jimmy SJ. Ren and Wei Wang and Jiawei Wang and Stephen Shaoyi Liao},
  title = {{Exploring The Contribution of Unlabeled Data in Financial Sentiment\textbackslash\{\}n Analysis}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2013},
  doi = {10.1609/aaai.v27i1.8459},
}
Figure