Abstract
With the proliferation of its applications in various industries, sentiment analysis by using publicly available web data has become an active research area in text classification during these years. It is argued by researchers that semi-supervised learning is an effective approach to this problem since it is capable to mitigate the manual labeling effort which is usually expensive and time-consuming. However, there was a long-term debate on the effectiveness of unlabeled data in text classification. This was partially caused by the fact that many assumptions in theoretic analysis often do not hold in practice. We argue that this problem may be further understood by adding an additional dimension in the experiment. This allows us to address this problem in the perspective of bias and variance in a broader view. We show that the well-known performance degradation issue caused by unlabeled data can be reproduced as a subset of the whole scenario. We argue that if the bias-variance trade-off is to be better balanced by a more effective feature selection method unlabeled data is very likely to boost the classification performance. We then propose a feature selection framework in which labeled and unlabeled training samples are both considered. We discuss its potential in achieving such a balance. Besides, the application in financial sentiment analysis is chosen because it not only exemplifies an important application, the data possesses better illustrative power as well. The implications of this study in text classification and financial sentiment analysis are both discussed.
Introduction
The application of sentiment analysis to financial data, such as corporate annual reports, has gained significant traction. However, the high cost of manual labeling—often requiring domain expertise like that of a Certified Public Accountant (CPA)—makes semi-supervised learning (SSL) an attractive alternative. Despite its potential, SSL has faced a long-standing debate due to observed "performance degradation," where adding unlabeled data reduces model accuracy. This study investigates this phenomenon by introducing vocabulary size as a critical experimental dimension, framing the problem as a challenge of balancing the bias-variance tradeoff.
Methodology
The researchers utilized a Multinomial Naive Bayes (MNB) classifier integrated with the Expectation-Maximization (EM) algorithm. The core of the experimental design involves systematically varying the vocabulary size (number of features) across different ratios of labeled and unlabeled data. This approach allows for a granular view of how feature selection impacts model performance when unlabeled data is introduced.
Data Description
The study analyzed Management’s Discussion and Analysis (MD&A) sections from SEC Form 10-K filings (2011-2012) for 10 diverse companies (e.g., Microsoft, Ford, Monsanto). The dataset comprised 5,245 labeled sentences from 2012 and 5,342 unlabeled sentences from 2011. Labels (Positive, Negative, Neutral) were double-checked by a professional CPA to ensure high-quality ground truth.
Results and Discussion
The experiments reveal that performance degradation is not an inherent property of unlabeled data but a consequence of suboptimal feature selection relative to the total data volume.
As shown in the figure above, there is consistently a "sweet spot"—a specific range of vocabulary sizes—where the inclusion of unlabeled data significantly boosts classification accuracy compared to labeled data alone. Conversely, outside this range, the same unlabeled data can lead to performance degradation. SSL models exhibit higher sensitivity to the bias-variance tradeoff than supervised models; a poorly chosen vocabulary size leads to more radical accuracy drops in the SSL context than in purely supervised models. This sensitivity underscores the importance of re-balancing features when transitioning from supervised to semi-supervised learning.
Conclusion
By re-balancing the bias-variance tradeoff through effective feature selection, unlabeled data can be reliably leveraged to enhance financial sentiment analysis. The study demonstrates that the "mystery" of performance degradation can be resolved by considering the interaction between data volume and vocabulary size. This finding suggests that feature selection frameworks should incorporate information from both labeled and unlabeled sets to achieve optimal performance, turning unlabeled data from a potential foe into a valuable asset.
Summary updated