Big Data in Finance

Itay Goldstein, Chester S. Spatt, Mao Ye

TL;DR

This paper provides a foundational framework for "Big Data" in finance, defining it through the pillars of large size, high dimensionality, and complex structure. By synthesizing a special issue of the Review of Financial Studies, the authors demonstrate how machine learning and unstructured datasets are moving the needle on classic problems like corporate governance, market microstructure, and asset pricing.

What Problem Does This Paper Address?

While "Big Data" is a buzzword in engineering and computer science, its specific utility in financial economics has been less clearly defined. Traditional financial research often relies on "small data"—structured, quarterly reports and linear regressions (OLS). However, these methods struggle to capture the non-linearities of modern markets or the nuances of "soft" information like corporate culture.

The authors address the gap between advanced computational tools and economic inquiry. They tackle the question: How does the transition from human-centric trading to machine-dominated markets change the way we measure liquidity, select corporate leaders, and regulate the financial system?

Methods

The paper serves as a meta-analysis of several cutting-edge methodologies applied to finance:

  • Non-linear Machine Learning: Moving beyond OLS to capture high-order interaction terms between variables, which is crucial for predicting complex outcomes like director performance.
  • Natural Language Processing (NLP): Utilizing word embeddings (which capture semantic meaning) rather than simple "bag-of-words" counts to quantify abstract concepts like corporate culture from earnings calls.
  • Matrix Completion: A machine learning technique used to interpolate missing data and latent factors in asset pricing to combat "data snooping" and the "factor zoo" (the tendency to find false-positive statistical patterns).
  • High-Frequency Econometrics: Applying ML to cross-asset microstructure data to see if traditional liquidity measures (like the Roll measure or Kyle’s $\lambda$) still hold up in an era of algorithmic trading.

Datasets

The paper highlights several massive or novel datasets that go beyond standard stock price feeds:

  • OATS (Order Audit Trail System): A comprehensive regulatory dataset from FINRA that includes broker identities, allowing researchers to identify agency conflicts.
  • Bitly Clickstream Data: Used as a proxy for "information demand" by tracking how many users click on shortened URLs for specific economic news.
  • Earnings Call Transcripts: Unstructured textual data used to map corporate culture onto five dimensions (innovation, integrity, quality, respect, and teamwork).
  • SEC Tick Size Pilot (TSP): Used as a controlled experiment to establish causality in broker routing behavior.
  • Alternative Data: The authors note the rising importance of satellite imagery, managerial vocal cues (audio), and credit card usage data.

Key Insights and Ideas

  • The "Psychology of Machines": As machines become the primary decision-makers in markets, we need a field of "algorithmic behavioral finance." Just as human psychology birthed behavioral finance, machine "biases" (embedded in their code) now drive market dynamics.
  • Homophily vs. Algorithms: Human boards often suffer from homophily—hiring people similar to themselves. ML models, by focusing on performance data rather than social networks, tend to select more diverse and higher-performing directors.
  • Quantifying the Intangible: Corporate culture is no longer a "vibe." Through NLP, it can be measured as a quantitative variable that predicts which firms are likely to be acquirers in M&A deals.
  • The Feedback Loop: Firms are beginning to "cater" to machines. For example, companies may adjust the language in their 10-K filings specifically to score better with the algorithms that read them.

Key Results

  • Director Selection: Algorithms can identify a pool of candidates who significantly outperform human-selected directors in out-of-sample shareholder approval tests.
  • Information Sensitivity: A one-standard-deviation increase in Bitly clicks on news related to nonfarm payrolls (NFP) raises the sensitivity of US Treasury yields by 4 to 6 basis points.
  • Broker Conflicts: Brokers who route orders to their own affiliated Alternative Trading Systems (ATSs) provide lower execution quality (lower fill rates and higher costs) for their customers.
  • Cross-Asset Dominance: While simple models work for predicting a single asset's liquidity, ML strictly dominates traditional statistics when trying to predict movements across different asset classes (e.g., using equity futures to predict bonds).

Limitations and Future Work

  • From Humans to Machines: Most current research uses ML to understand human behavior. The authors argue the next frontier is using ML to understand other machines.
  • Regulation Lag: Current regulations (like those for "odd lots" of less than 100 shares) were designed for humans. High-frequency traders now use these rules to "hide" their trades, necessitating a total rethink of regulatory frameworks.
  • Data Privacy and Fairness: As data becomes "the new oil," the authors highlight the urgent need for research into the ethics, ownership, and fair access to financial data.
  • Theoretical Gap: We have the tools to describe the world (ML), but we still need better economic theory to explain why these non-linear interactions occur.

Summary updated

Abstract

Big data is revolutionizing the finance industry and has the potential to significantly shape future research in finance. This special issue contains articles following the 2019 NBER/ RFS conference on big data. In this Introduction to the special issue, we define the "Big Data" phenomenon as a combination of three features: large size, high dimension, and complex structure. Using the articles in the special issue, we discuss how new research builds on these features to push the frontier on fundamental questions across areas in finance -including corporate finance, market microstructure, and asset pricing. Finally, we offer some thoughts for future research directions.
BibTeX
@misc{goldstein2021big,
  author = {Itay Goldstein and Chester S. Spatt and Mao Ye},
  title = {{Big Data in Finance}},
  year = {2021},
  doi = {10.2139/ssrn.3809447},
  howpublished = {SSRN},
}
Figure