Abstract
Estimating causal effects from observational data often requires accounting for confounders captured in text. Traditional "two-step" approaches—extracting topics and then performing regression—violate the Frisch-Waugh-Lovell theorem, leading to biased estimates. We present Bayesian Topic Regression (BTR), a joint generative model that simultaneously learns text representations and regression parameters. BTR provides theoretically grounded, unbiased causal inference and outperforms existing supervised topic models and deep learning benchmarks in both synthetic and real-world datasets.
Introduction
In social science and medicine, researchers frequently aim to estimate the causal effect of a treatment on an outcome using observational data. A primary challenge is confounding, where variables influence both the treatment and the outcome, creating spurious correlations. As text data (e.g., reviews, articles) increasingly serves as a proxy for these hidden confounders, the standard "two-step" approach—applying Latent Dirichlet Allocation (LDA) followed by regression—has become common.
However, this methodology is fundamentally flawed. According to the Frisch-Waugh-Lovell (FWL) theorem, if text features are correlated with other numerical confounders, they must be estimated jointly. Independent estimation results in systematic bias. Bayesian Topic Regression (BTR) addresses this by performing supervised representation learning for text features jointly with regression training.
Methodology
BTR is a fully Bayesian generative model that integrates a supervised topic model with a linear regression framework. Unlike standard models where topics are learned solely to describe text, BTR allows the outcome variable and numerical covariates to directly inform the topic discovery process.
To manage the complexity of joint estimation, BTR utilizes a Gibbs Expectation-Maximization (EM) algorithm. The E-step employs Gibbs sampling to identify latent topic assignments, offering statistical guarantees of exactness often absent in neural variational inference. The M-step updates regression parameters using standard Bayesian linear regression. This architecture ensures the model respects the FWL theorem by correctly partitioning the signal between text-derived topics and numerical covariates.
Experimental Evaluation
The model was evaluated across three distinct data regimes:
- Synthetic Data: 10,000 documents with known ground truth, where numerical features were intentionally correlated with specific text topics to test for bias.
- Semi-Synthetic Data: Real text from Yelp and Booking.com with "injected" causal effects to measure recovery accuracy.
- Real-World Data: Full datasets from Yelp and Booking.com used to predict customer ratings based on review text and business metadata.
Key Results
Bias Recovery
In synthetic tests, BTR was the only model to consistently recover true regression weights. Benchmark models, including sLDA and rSCHOLAR (a neural supervised topic model), exhibited significant bias when features were correlated.
Causal Accuracy
On semi-synthetic data, BTR’s estimate of the Treatment Effect (TE) remained remarkably close to the ground truth across varying topic counts. In contrast, other models drifted significantly as the number of topics changed.
Predictive Performance
On real-world Yelp data, BTR achieved a predictive $R^2$ of 0.630, outperforming complex RNN-based models (0.582) and traditional two-step topic models. This suggests that joint estimation not only aids causal inference but also improves the underlying text representation.
Discussion and Future Work
BTR demonstrates that joint estimation is a theoretical requirement for unbiasedness in the presence of text confounders. While the current framework assumes a linear causal effect and utilizes a bag-of-words representation, it provides a robust foundation for future extensions. Future research may explore integrating more sophisticated language representations, such as transformers, while maintaining the essential joint estimation requirement to ensure causal validity.
Summary updated