Abstract: Big data is revolutionizing the finance industry and has the potential to significantly shape future research in finance. This special issue contains articles following the 2019 NBER/ RFS conference on big data. In this Introduction to the special issue, we define the "Big Data" phenomenon as a combination of three features: large size, high dimension, and complex structure. Using the articles in the special issue, we discuss how new research builds on these features to push the frontier on fundamental questions across areas in finance—including corporate finance, market microstructure, and asset pricing. Finally, we offer some thoughts for future research directions.
Introduction: Defining the Big Data Era
While "Big Data" has long been a staple of computer science and engineering, its precise role in financial economics is still being mapped. Historically, financial research has relied on "small data"—structured quarterly reports and linear regressions (OLS). While effective for many years, these traditional methods often struggle to capture the non-linearities of modern markets or the nuances of "soft" information, such as corporate culture.
In this special issue introduction, authors Itay Goldstein, Mao Ye, and Chester S. Spatt define Big Data through three primary pillars: large size, high dimensionality, and complex structure. The transition from human-centric trading to machine-dominated markets necessitates a shift in how we measure liquidity, select corporate leaders, and regulate the financial system. This paper synthesizes research that bridges the gap between advanced computational tools and core economic inquiry.
The New Methodological Toolkit
The integration of Big Data into finance is driven by a suite of cutting-edge methodologies that move beyond traditional econometrics:
- Non-linear Machine Learning: Researchers are moving beyond OLS to capture high-order interaction terms. This is particularly vital for predicting complex outcomes, such as the future performance of corporate directors.
- Natural Language Processing (NLP): Modern NLP uses word embeddings to capture semantic meaning rather than simple "bag-of-words" counts. This allows for the quantification of abstract concepts like corporate culture from earnings call transcripts.
- Matrix Completion: This technique is used to interpolate missing data and identify latent factors in asset pricing. It serves as a robust defense against "data snooping" and the "factor zoo"—the proliferation of false-positive statistical patterns.
- High-Frequency Econometrics: By applying ML to cross-asset microstructure data, researchers can test whether traditional liquidity measures, such as the Roll measure or Kyle’s $\lambda$, remain valid in an era of algorithmic trading.
Novel Datasets and Information Demand
The scope of financial data has expanded far beyond standard stock price feeds. Researchers are now leveraging massive, unstructured, or alternative datasets:
- Regulatory Granularity: The Order Audit Trail System (OATS) from FINRA provides broker identities, enabling the identification of agency conflicts.
- Clickstream Data: Bitly click data serves as a proxy for "information demand," tracking how users engage with specific economic news.
- SEC Tick Size Pilot (TSP): This dataset is used as a controlled experiment to establish causality in broker routing behavior.
- Alternative Sources: Satellite imagery, managerial vocal cues (audio), and credit card usage data are becoming increasingly central to price discovery and firm valuation.
Key Insights: The Psychology of Machines
One of the most profound shifts is the emergence of the "Psychology of Machines." As algorithms become the primary decision-makers in markets, we are witnessing the birth of "algorithmic behavioral finance." Just as human psychology birthed behavioral finance, machine "biases"—embedded in their code or training data—now drive market dynamics.
Furthermore, Big Data is challenging human biases in corporate governance. While human boards often suffer from homophily (hiring similar individuals), ML models focusing on performance data tend to select more diverse and higher-performing directors. We also see a "feedback loop" where firms begin to "cater" to machines, adjusting the language in their 10-K filings specifically to score better with the algorithms that read them.
Empirical Findings
The research presented in this issue yields several striking results:
- Director Selection: Algorithms can identify candidate pools that significantly outperform human-selected directors in out-of-sample shareholder approval tests.
- Information Sensitivity: A one-standard-deviation increase in Bitly clicks on news related to nonfarm payrolls (NFP) raises the sensitivity of US Treasury yields by 4 to 6 basis points.
- Broker Conflicts: Brokers who route orders to their own affiliated Alternative Trading Systems (ATSs) often provide lower execution quality, characterized by lower fill rates and higher costs for customers.
- Cross-Asset Dominance: While simple models may suffice for single-asset liquidity prediction, ML strictly dominates traditional statistics when predicting movements across different asset classes (e.g., using equity futures to predict bonds).
Future Directions and Regulatory Challenges
The transition from studying human behavior to studying machine behavior represents the next frontier. However, this shift brings significant challenges:
- Regulatory Lag: Current regulations (e.g., rules for "odd lots" of less than 100 shares) were designed for human traders. High-frequency traders can exploit these rules to "hide" trades, necessitating a total rethink of regulatory frameworks.
- Ethics and Privacy: As data becomes "the new oil," research into the ethics, ownership, and fair access to financial data is urgent.
- Theoretical Integration: While ML provides the tools to describe the world with high precision, we still require robust economic theory to explain why these complex, non-linear interactions occur.
By embracing Big Data, the finance profession can move toward a more nuanced, real-time understanding of the global economy.
Summary updated