Mining the Web for the "Voice of the Herd" to Track Stock Market Bubbles

Aaron Gerow, Mark T. KeaneIJCAI 2011 · DOI

TL;DR

This paper introduces a novel method for identifying stock market bubbles by analyzing the linguistic patterns in online financial news. By applying power-law analysis to word frequencies in a large corpus of financial articles, the researchers found that emergent structure in language, specifically verb convergence, accurately tracks market movements and predicts bubble formation.

What Problem Does This Paper Address?

Traditional methods for identifying stock market bubbles primarily rely on analyzing price movements and volatility. However, these methods can be retrospective or explained away by new valuation models. The authors propose an alternative approach inspired by John D. Rockefeller's observation that bubbles occur when everyone has the same positive view of the market. They aim to capture this "voice of the herd" by analyzing the language used by financial commentators, hypothesizing that during a bubble, the language will exhibit emergent structure and increased agreement, reflecting a collective, positive sentiment.

Methods

The core methodology involves applying power-law analysis to linguistic features extracted from a corpus of financial news articles. Specifically, the researchers:

  • Corpus Creation: Gathered over 17,000 online finance-related articles from the Financial Times, New York Times, and BBC over a four-year period.
  • Linguistic Analysis: Used natural language processing techniques (shallow parsing, lemmatization, part-of-speech tagging) to extract verb-phrases and noun-phrases.
  • Power-Law Analysis: Modeled the frequency distribution of these linguistic units using a generalized power-law form ($y = Cx^{-\alpha}$). The exponent ($\alpha$) of this distribution is a key metric.
  • Time-Series Analysis: Calculated weekly power-law distributions and tracked changes in the exponent ($\alpha$) over time. They used an 8-week moving average of $\alpha$ to smooth out weekly fluctuations and capture emergent trends.
  • Valency Analysis: Employed a symmetric Kullback-Leibler (K-L) divergence metric to measure shifts in the valence (positive, negative, neutral) of lemma-object pairs (verb-noun combinations) over time.

Datasets

  • Corpus: 17,713 online, finance-related articles from the Financial Times, the New York Times, and the BBC, totaling over 10 million words. The corpus covers a four-year period from January 1st, 2006, to January 1st, 2010.
  • Stock Market Data: Weekly closing levels for the Dow Jones Industrial Average (DJI), the FTSE-100, and the NIKKEI-225 were used for comparison.

Key Insights and Ideas

  • Language as a Leading Indicator: The study demonstrates that linguistic patterns in financial news can serve as a leading indicator for stock market movements, particularly for identifying bubble periods.
  • "Voice of the Herd" Manifests in Language: During bubble periods, the language of financial commentators shows emergent structure, characterized by increased agreement and a convergence towards positive sentiment.
  • Power Laws in Language: The frequency distributions of verb-phrases and noun-phrases in financial news follow power laws, and the exponents of these laws change predictably with market conditions.
  • Verb Convergence as a Bubble Signal: A key finding is that the convergence of verb usage (i.e., a smaller set of verbs being used more frequently) is a strong indicator of a market bubble. This suggests a narrowing of focus and increasing uniformity in reporting.
  • Positive Sentiment Spike: The K-L divergence analysis of verb valency reveals a marked divergence in positive language usage leading up to the 2007 crash, indicating a surge in optimistic reporting.

Key Results

  • Strong Correlations: The 8-week moving average of the power-law exponent for verb distributions showed high and positive correlations with the DJI (r = .79), FTSE-100 (r = .78), and NIKKEI-225 (r = .73).
  • Bubble Tracking: The model's values peaked almost simultaneously with the DJI during the 2007 bubble, and then fell precipitously during the crash. Similarly, the model's values began to rise again before the DJI reached its lowest point in March 2009.
  • Noun-Phrase Correlations: Noun-phrase analysis also revealed positive correlations with the indices, though slightly weaker than verb-phrases (DJI: r = .65, FTSE-100: r = .62, NIKKEI-225: r = .62).
  • Valency Shift: Leading up to the 2007 crash, positive lemma-object pairs (e.g., "stocks rallied") showed a marked divergence from negative ones, indicating a shift towards overwhelmingly positive language.

Limitations and Future Work

  • Correlation vs. Causation: The authors acknowledge that the correlations, while strong, are indicative rather than conclusive, given the non-stationary nature of the data.
  • Artifacts of Method: To address concerns that the results might be artifacts of the chosen analysis units, they performed analyses on arbitrary units (e-delimited strings and every third word). These arbitrary analyses yielded low-to-moderate, negative correlations, suggesting that the positive correlations found with verb and noun phrases are not merely methodological artifacts.
  • Limited Scope of Valency: While the valency analysis provided supporting evidence, the authors note that it's harder to analyze than verb convergence and that the neutral category often served as a default.
  • Future Work: The paper suggests that this language-based measure can serve as an independent tool to complement existing pricing analyses for assessing impending market events. The authors also note that further research could explore the nuances of language shifts, particularly during periods of economic recovery or crisis, as seen in the footnote regarding the US Congress's bailout package formulation.

Summary updated

BibTeX
@inproceedings{gerow2011mining,
  author = {Aaron Gerow and Mark T. Keane},
  title = {{Mining the Web for the "Voice of the Herd" to Track Stock Market Bubbles}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2011},
  doi = {10.5591/978-1-57735-516-8/ijcai11-374},
}
Figure