Abstract: This project investigates the interplay of technical, market, and statistical factors in predicting stock market performance, with a primary focus on S&P 500 companies. Utilizing a comprehensive dataset spanning multiple years, the analysis constructs advanced financial metrics, such as momentum indicators, volatility measures, and liquidity adjustments. The machine learning framework is employed to identify patterns, relationships, and predictive capabilities of these factors. The integration of traditional financial analytics with machine learning enables enhanced predictive accuracy, offering valuable insights into market behavior and guiding investment strategies. This research highlights the potential of combining domain-specific financial expertise with modern computational tools to address complex market dynamics.
Introduction
Forecasting stock returns remains one of the most significant challenges in financial engineering due to the stochastic nature of markets and the non-linear relationships between drivers. Traditional linear models often fail to capture the complexity of the "Factor Zoo"—the vast array of potential predictors available to modern analysts. This paper proposes an AI-enhanced framework that bridges the gap between traditional financial factor analysis and machine learning to improve predictive accuracy and portfolio construction for S&P 500 constituents.
Data and Feature Engineering
The study utilizes high-quality stock-level data from the Center for Research in Security Prices (CRSP) database, accessed via the Wharton Research Library. The scope is limited to S&P 500 constituent companies from January 1, 2019, through the end of 2023. The raw data includes daily prices, returns, trading volumes, and shares outstanding.
From this raw data, the researchers engineered 32 distinct financial factors. These factors are categorized into four primary domains: size (Market Capitalization), momentum, liquidity (e.g., Amihud Illiquidity, Turnover Ratio), and volatility.
Methodology: The Two-Layer Filtering Logic
To address the risk of overfitting and the redundancy inherent in a 32-factor model, a novel two-layer feature filtering process was implemented:
- Layer 1: Target Correlation Filtering: Features were initially evaluated based on their direct correlation with stock returns. By setting a specific correlation threshold, the initial set was reduced from 32 to 22 factors. This step ensures that only features with a statistically significant relationship to the target variable are retained.
- Layer 2: Redundancy and Performance Optimization: The remaining factors were screened for multicollinearity. Any pair of features with a correlation coefficient exceeding 0.75 was flagged, and redundant variables were removed. Following this, a Random Forest Regressor was utilized to test all permutations of the remaining factors, ultimately selecting the top 10 features that maximized the model's $R^2$ score.
Results and Analysis
Model Comparison
The researchers compared four distinct modeling approaches: Linear Regression, Ridge Regression, Random Forest, and Gradient Boosting. The Gradient Boosting Regressor (GBR) emerged as the superior model, achieving the highest $R^2$ (0.228) and the lowest Mean Squared Error (0.00648).
A critical observation was the dramatic drop in $R^2$ values following the filtering process (from near-perfect scores of ~0.98 down to ~0.20). While numerically lower, the authors argue this represents a significant improvement in model robustness. The high initial $R^2$ was indicative of the model "memorizing" market noise; the filtered model, conversely, captures genuine, generalizable signals.
Key Factor Insights
Using SHAP (SHapley Additive exPlanations) values, the study identified "Momentum Liquidity" as the most influential predictor. This interaction term suggests that price trends are most predictive when adjusted for the ease with which a stock can be traded. Secondary influential features included "Volatility Dynamics" and "High-Low Spread," confirming that risk measures are indispensable for return forecasting.
Portfolio Backtesting
A backtesting strategy using a 36-month rolling window was employed to select the top 100 stocks for an equally weighted portfolio. This strategy yielded a Sharpe Ratio of 0.25. While the portfolio demonstrated lower volatility than the broader market and a steady upward trend, it underperformed the S&P 500 benchmark in raw cumulative returns. This suggests the model prioritized risk-adjusted stability over aggressive growth during the specific testing window.
Conclusion and Future Work
This research demonstrates that integrating machine learning with rigorous financial feature engineering can distill complex market data into actionable insights. While the current model provides a stable foundation, future iterations will look to incorporate macroeconomic indicators (interest rates, inflation) and Natural Language Processing (NLP) for sentiment analysis. Additionally, moving toward deep learning architectures like Long Short-Term Memory (LSTM) networks may better capture the temporal dependencies inherent in financial time-series data.
Summary updated