Abstract
Economists have long posited that efficient markets incorporate all available information into prices. However, the specific dynamics of this process remain poorly understood. This paper provides a theoretical and empirical framework for understanding how information is integrated over time. By analyzing political betting markets, we demonstrate that market accuracy improves at a predictable rate and present a semi-automated system that leverages market volatility to detect and explain major news events using natural language processing (NLP).
Introduction
The Efficient Market Hypothesis suggests that prices are the best possible estimates of future outcomes. Yet, the process of how information is incorporated over time is often treated as a "black box." This research addresses two primary challenges: mathematically proving that market accuracy improves predictably as an event approaches, and utilizing the "wisdom of the crowd" as a sensor to automatically explain the causes of significant market shifts.
Theoretical Framework and Methods
The study approaches market dynamics through three distinct lenses: theoretical proofs, simulation, and automated text analysis. Using the "Forecast Accuracy" assumption—the idea that current prices are the best possible estimate of future outcomes—we derive theorems showing that in a world where an event occurs, the price is expected to rise by an amount proportional to the variance of the price.
To simulate this, we employ a "Coin-Flip Model," where an event is represented by a series of $n$ information units (coin flips). As each unit is revealed, the market updates its probability. This model produces price curves remarkably similar to real-world data from the Iowa Electronic Market (IEM). Finally, we identify "pivotal dates" where the log-likelihood of a market price changes significantly and compare text corpuses from before and after these dates using Expected Entropy Loss (Information Gain) to extract semantic explanations.
Information Incorporation and Accuracy
Empirical analysis of 22 political markets on the IEM reveals that the "logarithmic score"—a measure of market accuracy—increases roughly linearly over time. This suggests that, on average, information flows into these political markets at a relatively constant rate, rather than in unpredictable bursts. This linear trend provides a baseline for understanding how markets converge on the truth as an event's conclusion nears.
Event Detection and Semantic Explanation
Beyond tracking general trends, markets can serve as high-signal filters for news. Significant price swings often act as triggers, telling an algorithm exactly when to look at the news to find a "semantic explanation" for a shift in public sentiment. By monitoring these spikes, we can bypass the noise of general news cycles and focus on the specific information that moved the needle.
Experimental Results
The research yields several key insights and validations:
- The "True World" Drift: While an efficient market is a martingale, in the "world" where the event eventually happens, the price statistically trends upward at a rate proportional to market volatility. The authors found a close fit between these theoretical predictions and real IEM data.
- Automated Explanations: The entropy-loss algorithm successfully identified the "why" behind major market moves. In the 2000 NY Senate race, the top extracted features during price drops were "cancer" and "prostate" (matching Giuliani’s health announcement) and later "Rick Lazio" (his replacement). Similarly, a spike in the "Extra-terrestrial life" market correctly pulled keywords like "meteorite" and "Martian" following NASA's 1996 announcement regarding potential life on Mars.
Conclusion and Future Work
This work demonstrates that markets are active sensors of information flow. While the current models assume discrete, equal-sized information chunks, future work will address non-linear "shocks" and refine NLP techniques to reduce source-specific noise. Applying these methods to broader, noisier financial markets remains a significant challenge but offers the potential for automated, real-time news interpretation.
Summary updated