Bayesian Topic Regression for Causal Inference

Maximilian Ahrens, Julian Ashwin, Jan-Peter Calliess, Vu NguyenEMNLP 2021 · DOI

Abstract

Estimating causal effects from observational data often requires accounting for confounders captured in text. Traditional "two-step" approaches—extracting topics and then performing regression—violate the Frisch-Waugh-Lovell theorem, leading to biased estimates. We present Bayesian Topic Regression (BTR), a joint generative model that simultaneously learns text representations and regression parameters. BTR provides theoretically grounded, unbiased causal inference and outperforms existing supervised topic models and deep learning benchmarks in both synthetic and real-world datasets.

Introduction

In social science and medicine, researchers frequently aim to estimate the causal effect of a treatment on an outcome using observational data. A primary challenge is confounding, where variables influence both the treatment and the outcome, creating spurious correlations. As text data (e.g., reviews, articles) increasingly serves as a proxy for these hidden confounders, the standard "two-step" approach—applying Latent Dirichlet Allocation (LDA) followed by regression—has become common.

However, this methodology is fundamentally flawed. According to the Frisch-Waugh-Lovell (FWL) theorem, if text features are correlated with other numerical confounders, they must be estimated jointly. Independent estimation results in systematic bias. Bayesian Topic Regression (BTR) addresses this by performing supervised representation learning for text features jointly with regression training.

Methodology

BTR is a fully Bayesian generative model that integrates a supervised topic model with a linear regression framework. Unlike standard models where topics are learned solely to describe text, BTR allows the outcome variable and numerical covariates to directly inform the topic discovery process.

To manage the complexity of joint estimation, BTR utilizes a Gibbs Expectation-Maximization (EM) algorithm. The E-step employs Gibbs sampling to identify latent topic assignments, offering statistical guarantees of exactness often absent in neural variational inference. The M-step updates regression parameters using standard Bayesian linear regression. This architecture ensures the model respects the FWL theorem by correctly partitioning the signal between text-derived topics and numerical covariates.

Experimental Evaluation

The model was evaluated across three distinct data regimes:

  1. Synthetic Data: 10,000 documents with known ground truth, where numerical features were intentionally correlated with specific text topics to test for bias.
  2. Semi-Synthetic Data: Real text from Yelp and Booking.com with "injected" causal effects to measure recovery accuracy.
  3. Real-World Data: Full datasets from Yelp and Booking.com used to predict customer ratings based on review text and business metadata.

Key Results

Bias Recovery

In synthetic tests, BTR was the only model to consistently recover true regression weights. Benchmark models, including sLDA and rSCHOLAR (a neural supervised topic model), exhibited significant bias when features were correlated.

Figure 2: BTR (top left) successfully recovers true regression weights (red dots) within its estimated intervals, whereas competing models fail.

Causal Accuracy

On semi-synthetic data, BTR’s estimate of the Treatment Effect (TE) remained remarkably close to the ground truth across varying topic counts. In contrast, other models drifted significantly as the number of topics changed.

Figure 3: BTR's estimated treatment effect (blue line) tracks the ground truth (red dashed line) more accurately than any benchmark across different topic counts.

Predictive Performance

On real-world Yelp data, BTR achieved a predictive $R^2$ of 0.630, outperforming complex RNN-based models (0.582) and traditional two-step topic models. This suggests that joint estimation not only aids causal inference but also improves the underlying text representation.

Discussion and Future Work

BTR demonstrates that joint estimation is a theoretical requirement for unbiasedness in the presence of text confounders. While the current framework assumes a linear causal effect and utilizes a bag-of-words representation, it provides a robust foundation for future extensions. Future research may explore integrating more sophisticated language representations, such as transformers, while maintaining the essential joint estimation requirement to ensure causal validity.

Summary updated

BibTeX
@inproceedings{ahrens2021bayesian,
  author = {Maximilian Ahrens and Julian Ashwin and Jan-Peter Calliess and Vu Nguyen},
  title = {{Bayesian Topic Regression for Causal Inference}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.644},
}
Figure