Automatic Domain Adaptation Outperforms Manual Domain Adaptation for Predicting Financial Outcomes

Marina Sedinkina, Nikolas Breitkopf, Hinrich Schütze

Abstract

In this paper, we automatically create sentiment dictionaries for predicting financial outcomes. We compare three approaches: (I) manual adaptation of the domain-general dictionary H4N, (ii) automatic adaptation of H4N and (iii) a combination consisting of first manual, then automatic adaptation. In our experiments, we demonstrate that the automatically adapted sentiment dictionary outperforms the previous state of the art in predicting the financial outcomes excess return and volatility. In particular, automatic adaptation performs better than manual adaptation. In our analysis, we find that annotation based on an expert's a priori belief about a word's meaning can be incorrect -- annotation should be performed based on the word's contexts in the target domain instead.

Introduction

In financial Natural Language Processing (NLP), the "tone" of company filings, such as the SEC's 10-K annual reports, is a critical indicator for investors. However, general-purpose sentiment dictionaries like the Harvard-IV-4 (H4N) often fail in this domain because words like "liability" or "cost" are standard financial terminology rather than inherently negative indicators. The current industry standard, the Loughran and McDonald (L&M) dictionary, addresses this through manual expert reclassification. However, manual processes are slow, subjective, and often overlook the specific contextual nuances of the target domain. This research explores whether machine learning can automatically adapt these dictionaries to outperform human-expert classifications.

Methodology

The researchers developed a pipeline for Automatic Domain Adaptation:

  1. Word Embeddings: A Word2vec (CBOW) model was trained on a corpus of 206,790 10-K filings (1994–2013) to capture the domain-specific meanings of words.
  2. Classifier Training: Support Vector Machines (SVMs) were trained using the original H4N labels to learn the relationship between embeddings and sentiment categories.
  3. Adaptation Strategies:
  4. Relabeling (RE): Existing H4N words were reclassified based on their financial context embeddings.
  5. Adding (ADD): New words from the 10-K corpus were identified and categorized by the SVM.
  6. Evaluation: Ordinary Least Squares (OLS) regression was used to predict excess return and volatility, controlling for standard financial variables like firm size and book-to-market ratio.

Results

Dictionary Composition

The automatically adapted dictionary ($H4N_{RE}$) is significantly more concise than its manual counterparts. While the original H4N contains over 4,000 words, the refined $H4N_{RE}$ focuses on a core set of 338 highly predictive terms. This suggests that many words experts intuitively label as "negative" may actually introduce noise that leads to overfitting.

Table 1: Dictionary sizes for various adaptation strategies.

Predictive Performance

The automatic adaptation strategy demonstrated superior predictive power for financial outcomes. For excess returns, $H4N_{RE}$ achieved a standardized coefficient of -0.229, nearly triple the magnitude of the manual L&M dictionary's -0.080. A higher absolute value here indicates that the automatically generated dictionary is a much stronger predictor of stock price drops.

Table 2: Regression results for excess returns comparing manual (L&M) and automatic ($H4N_{RE}$) dictionaries.

In volatility prediction, $H4N_{RE}$ similarly outperformed manual dictionaries. When both manual and automatic dictionaries were included in the same regression model, the manual variables often lost statistical significance, proving that the automatic method captures all the useful information of the manual one and then some.

Discussion: Context vs. Intuition

The success of automatic adaptation lies in its ability to capture context over human intuition. For instance, experts might label "God" as positive, but in 10-K filings, it almost exclusively appears in "act of God" clauses, serving as a risk indicator. Similarly, the word "abeyance" is typically categorized as "uncertain," but its proximity to legal terms like "hearings" in financial embeddings correctly shifts it to a "litigious" category. This highlights that annotation should be performed based on a word's context in the target domain rather than an expert's a priori belief.

Conclusion

This study demonstrates that automatic domain adaptation using word embeddings and machine learning creates more robust and predictive sentiment dictionaries than manual expert classification. By focusing on how words are actually used in context, researchers can build tools that provide significantly higher incremental value for financial forecasting. Future work will explore temporal shifts in language to account for evolving financial terminology over time.

Summary updated

Abstract

In this paper, we automatically create sentiment dictionaries for predicting financial outcomes. We compare three approaches: (I) manual adaptation of the domain-general dictionary H4N, (ii) automatic adaptation of H4N and (iii) a combination consisting of first manual, then automatic adaptation. In our experiments, we demonstrate that the automatically adapted sentiment dictionary outperforms the previous state of the art in predicting the financial outcomes excess return and volatility. In particular, automatic adaptation performs better than manual adaptation. In our analysis, we find that annotation based on an expert's a priori belief about a word's meaning can be incorrect -- annotation should be performed based on the word's contexts in the target domain instead.
BibTeX
@inproceedings{sedinkina2019automatic,
  author = {Marina Sedinkina and Nikolas Breitkopf and Hinrich Schütze},
  title = {{Automatic Domain Adaptation Outperforms Manual Domain Adaptation for Predicting Financial Outcomes}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=ihocxts7eBw},
}
Figure