Abstract
Marina Sedinkina, Nikolas Breitkopf, Hinrich Schütze. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.
2019.
Introduction
In financial analysis, the "tone" of a company's annual report (the 10-K filing) is a powerful predictor of future performance. Traditionally, researchers used the Harvard Psychosociological Dictionary (H4N) to identify negative words. However, H4N is domain-general; words like "tax," "cost," or "liability" are labeled as negative in a general context, but they are neutral, everyday terms in finance.
To fix this, the current "gold standard" has been manual adaptation—experts like Loughran and McDonald (L&M) spent years manually reclassifying these words for the financial domain. This paper challenges that standard, asking: Can an algorithm, looking only at how words are used in context, outperform a human expert's a priori beliefs about what words mean?
Methods
The authors propose a pipeline for Automatic Domain Adaptation:
- Word Embeddings: They trained a Word2Vec (CBOW) model on a corpus of over 60,000 10-K filings to learn the "meaning" of words based on their financial context.
- Classification: They used Support Vector Machines (SVMs) trained on the original H4N labels to reclassify words.
- RE (Reclassification): They took the general H4N dictionary and used the SVM to decide if a word was truly "negative" or "non-negative" based on its financial embedding.
- ADD (Addition): They used the SVM to find new words in the H4N list that human experts (L&M) had missed but the model flagged as relevant to financial sentiment.
- Evaluation: They used Ordinary Least Squares (OLS) regression to see how well these different dictionaries predicted two key financial outcomes: excess return (stock performance above the market average) and volatility (price fluctuations).
Datasets
- SEC EDGAR: The primary source was 206,790 10-K filings from 1994 to 2013.
- CRSP: Stock price data used to calculate financial outcomes.
- Final Sample: After filtering for firms with available market data and sufficient text length, the researchers used a corpus of 60,432 10-Ks.
- Dictionaries: The Harvard-IV-4 TagNeg (H4N) served as the source, while the Loughran and McDonald (L&M) dictionary served as the manual baseline.
Key Results
The study found that automated adaptation significantly improves the predictive power of sentiment dictionaries compared to manual reclassification.
Predicting Excess Return
In predicting excess return, the automatically created $H4N_{RE}$ dictionary had a standard coefficient (-0.229) nearly three times larger than the manual L&M dictionary (-0.080).
Predicting Volatility
For predicting volatility, $H4N_{RE}$ again outperformed the manual version, with an explanatory value (standard coefficient of 0.173) more than triple that of the L&M baseline (0.047).
Quantitative analysis showed that only 12% of the words humans labeled as "negative" actually survived the automatic reclassification. This suggests that many words in manual dictionaries are "noise" that doesn't actually help predict financial outcomes.
Discussion
- Context Trumps Intuition: Human annotators often label words based on their "dictionary definition" rather than their usage. For example, humans might label "God" as positive. However, in 10-Ks, "God" almost exclusively appears in the phrase "act of God" (referring to disasters/legal liabilities). The algorithm correctly clustered "God" with terms like "terrorism" and "war," recognizing its negative financial implication.
- The "Abeyance" Case: Humans labeled "abeyance" as an "uncertainty" word. The algorithm labeled it as "litigious." Looking at the data, "abeyance" is almost always used in legal contexts in 10-Ks (e.g., "appeal in abeyance"), proving the algorithm captured the domain-specific nuance better than the human.
- Leaner is Meaner: The automatically adapted dictionary ($H4N_{RE}$) is much smaller than the manual L&M dictionary (338 words vs. 2,355). By focusing only on frequent, high-signal words, the model is more robust and less prone to overfitting.
Limitations and Future Work
- Temporal Shifts: The authors acknowledge that the meaning of words can change over time. A word that was neutral in 1994 might become "negative" in 2013. Future work could involve dynamic embeddings that track linguistic shifts across decades.
- Corpus Size: While 60,000 documents is substantial, the authors suggest that training on even larger, more diverse financial corpora could further refine the sentiment signals.
- Interpretability vs. Complexity: While this method uses "interpretable" OLS regression, there is a trade-off. Future research could explore if more complex deep learning architectures provide better predictions without sacrificing the explainability required by the finance industry.
Summary updated