TL;DR
This paper proposes a robust Bayesian modeling framework using Gaussian Process Regression on Reddit data to predict the Consumer Confidence Index (CCI). The authors demonstrate that this approach can reliably estimate CCI months in advance, significantly reducing the need for frequent traditional surveys and offering superior accuracy compared to existing methods.
What Problem Does This Paper Address?
Traditional surveys, like those used to measure the Consumer Confidence Index (CCI), are expensive, time-consuming, and face declining public participation. While social media data has been explored as a cheaper, more frequent alternative, previous attempts to predict CCI using this data have shown diminishing returns with newer datasets. This paper revisits the potential of social media data, specifically Reddit, to accurately forecast CCI, aiming to reduce survey costs by enabling less frequent polling, with periodic surveys only needed for model calibration.
Methods
The core of the proposed methodology is a Gaussian Process Regression (GPR) model, a Bayesian non-parametric approach known for its flexibility in modeling non-linear relationships and its ability to provide uncertainty estimates. The authors leverage this framework to model the relationship between longitudinal covariates extracted from Reddit and the target CCI.
Key methodological contributions include:
- Data Preprocessing and Feature Extraction:
- Content Extraction: Using
doc2vecembeddings to represent text from Reddit posts, arguing that content features alone can capture overall consumer sentiment trends.
- Content Extraction: Using
- Sentiment Extraction: Employing VADER for sentiment analysis and LIWC for categorizing words into various linguistic and psychological components.
- Handling Noisy Data: Implementing a clustering approach (K-means) on post embeddings to identify and remove irrelevant clusters (e.g., advertisements, non-English content, bots).
- Trustworthy User Identification: Training a logistic regression classifier to assign weights to users based on their activity and history, giving more importance to "trustworthy" users.
- Longitudinal Covariate Creation: Aggregating post features (weighted by user trust scores) to create daily and monthly covariate time series.
- Gaussian Process Regression with Matern Kernel: Utilizing a Matern 3/2 kernel, which is well-suited for learning non-smooth behaviors often seen in financial time-series data.
- Principled Experimentation Methodology:
- Prediction Step ($\Delta$): Investigating how model accuracy changes when predicting CCI at future time points.
- Correspondence Lag ($\alpha$): Examining if Reddit data leads or lags survey data, finding that direct alignment (no lag) yields the best results.
- Reducing Survey Frequency: Developing a strategy (using the
MonitorWithMissingTargetsroutine) to estimate CCI during periods when survey data is unavailable, by feeding model estimates back into the training data.
Datasets
- Reddit Data: Collected from the "economy," "finance," and "jobs" subreddits. Data spanned from March 2008 to August 2019.
- Consumer Confidence Index (CCI): Specifically, the Index of Consumer Sentiment (ICS) from the University of Michigan. Daily ICS data was available from January 2008 to May 2017, and monthly ICS data from March 2008 to August 2019.
Key Insights and Ideas
- Reddit as a Rich Source for Economic Signals: The paper strongly suggests that discussions on Reddit, particularly in economy-related subreddits, contain valuable signals for predicting consumer confidence, going beyond simple sentiment analysis.
- Content Features Matter: Contrary to some prior work focusing solely on sentiment, the authors demonstrate that text content features from Reddit posts are crucial for capturing overall consumer sentiment trends.
- Trustworthy Users Enhance Model Robustness: Incorporating a measure of user trustworthiness helps to filter out noise and improve the reliability of the extracted signals.
- GPR's Power for Non-linear Economic Forecasting: Gaussian Process Regression proves to be a powerful tool for modeling the complex, non-linear relationship between social media activity and economic indicators, while also providing crucial uncertainty estimates.
- Significant Reduction in Survey Frequency Possible: The core contribution is showing that by using Reddit data, the frequency of costly traditional surveys can be substantially reduced (e.g., to every 2-4 months) without a drastic loss in predictive accuracy.
Key Results
- Accurate Long-Term Predictions: The model can reliably estimate daily and monthly CCI at least several months in advance, with performance metrics (RMSE, MAE) significantly outperforming baseline methods. For example, with a 28-day smoothing window and a prediction step of 28 days, the model achieved an RMSE of 5.52 and MAE of 4.23.
- Superior to Baselines: The proposed Reddit-based model significantly outperforms a "time-only" baseline (using only past CCI values) and single-feature baselines (using only sentiment or content).
- Reduced Survey Costs: Experiments simulating reduced survey availability (e.g., every 2nd, 3rd, or 4th month) showed that the model could maintain reasonable accuracy, demonstrating the potential for cost savings. For instance, with surveys every second month, the RMSE was 4.03.
- No Significant Lag: Correspondence lag analysis indicated that Reddit data and survey data are temporally aligned, suggesting that social media chatter reflects current sentiment rather than lagging behind it.
Limitations and Future Work
- Representativeness of Reddit Users: The authors acknowledge that Reddit users may not be fully representative of the general population, and the lack of demographic information in social media data can introduce bias.
- Sampling Bias: Datasets derived from social media platforms can suffer from sampling bias.
- Supplement, Not Replace: While the model can supplement traditional surveys and reduce their frequency, it may not completely replace them.
- Future Work: The authors plan to incorporate Twitter data into their model and expand their methodology to other survey data to better understand when social media data can substitute for or complement existing polling methods.
Summary updated