Abstract: We use quantitative media (blogs, and news as a comparison) data generated by a large-scale natural language processing (NLP) text analysis system to perform a comprehensive and comparative study on how company-related news variables anticipate or reflect the company's stock trading volumes and financial returns. Building on our findings, we give a sentiment-based market-neutral trading strategy which gives consistently favorable returns with low volatility over a long period. Our results are significant in confirming the performance of general blog and news sentiment analysis methods over broad domains and sources. Moreover, several remarkable differences between news and blogs are also identified.
Introduction
The Efficient Market Hypothesis (EMH) posits that stock prices instantaneously reflect all available information, theoretically precluding consistent market outperformance. However, human reactions to information are rarely instantaneous or purely rational. The primary obstacle to testing this in the modern era is the sheer volume of data; quantifying the collective "mood" of millions of articles and blog posts requires significant computational power. This study addresses the "Big Data" challenge of sentiment analysis by comparing traditional journalism with emerging social media platforms to determine which provides a more effective signal for financial traders.
Methodology
The researchers employed Lydia, a high-speed NLP pipeline designed for large-scale text analysis, to process approximately one terabyte of media data spanning 2005 to 2009. The methodology involved three distinct phases:
- Sentiment Extraction: Lydia identified specific entities (companies) within the text and calculated two primary metrics: Polarity (the balance of favorable versus unfavorable mentions) and Subjectivity (the density of sentiment-bearing words relative to neutral ones).
- Correlation Analysis: These linguistic variables were mapped against financial indicators, including trading volume, price fluctuations, and "abnormal returns" (performance relative to the broader market).
- The Trading Agent: A market-neutral strategy was designed to isolate the predictive power of sentiment. Each day, the system ranked companies by sentiment, "going long" (buying) the top $n$ positive stocks and "shorting" (betting against) the bottom $n$ negative stocks in equal dollar amounts. This approach hedges against general market volatility, focusing strictly on the alpha generated by sentiment signals.
Sentiment Dynamics and Market Capitalization
A key finding of the study is the strong correlation (>0.4) between the frequency of company mentions and trading volume. Furthermore, "subjectivity"—the degree to which coverage is opinionated—also correlates with higher trading volume, suggesting that media controversy is a significant driver of market activity.
However, the effectiveness of these signals is highly dependent on company size. Sentiment analysis proved significantly more accurate for large-cap companies, which receive more intensive and consistent media coverage. This creates a cleaner statistical signal compared to the "thinly traded" small-cap stocks, where data sparsity can lead to noise.
Additionally, the researchers identified a "Recency Gap" between media types. News sentiment exhibits a sharp recency effect, impacting the market immediately before the signal vanishes. In contrast, sentiment from blogs and Twitter proved more persistent, with effects lasting two to three days, suggesting that while news breaks a story, social media reflects the slower-moving evolution of public opinion.
Performance and Diversification
The sentiment-based trading strategy yielded favorable returns across most of the study period, including the 2008 financial crisis. By successfully shorting stocks that the media viewed most pessimistically, the strategy remained profitable even during a general market collapse.
One critical observation was the "Diversification Decay": as the number of stocks ($n$) in the portfolio increased, the yearly return decreased. This indicates that the strongest predictive edge is concentrated at the extreme ends of the sentiment spectrum.
Conclusion and Limitations
While the results are promising, several real-world challenges remain. The simulation did not account for transaction costs, such as commissions and slippage, which can significantly erode profits in high-frequency daily trading. Furthermore, the 12-hour lag in the "spidering" process used for news retrieval highlights the difficulty of real-time execution. Future research may refine these strategies by filtering for specific market sectors, such as Life Insurance or Financial Services, which demonstrated the highest sensitivity to sentiment signals.
Summary updated