Adapting Text Embeddings for Causal Inference

Victor Veitch, Dhanya Sridhar, David M. BleiUAI 2020 · OpenReview

Abstract

Does adding a theorem to a paper affect its chance of acceptance? Does labeling a post with the author’s gender affect the post popularity? This paper develops a method to estimate such causal effects from observational text data by learning "causally sufficient embeddings." By adapting modern NLP tools like BERT and Topic Models to simultaneously model language and predict both treatments and outcomes, the authors provide a way to adjust for high-dimensional text confounders that would otherwise bias causal estimates.

The Challenge of Text Confounders

In causal inference, the primary goal is to isolate the effect of a treatment ($T$) on an outcome ($Y$). However, in observational studies, this relationship is often obscured by confounders ($W$)—variables that influence both the treatment assignment and the resulting outcome. When these confounders are embedded within unstructured text, such as the abstract of a scientific paper or the content of a social media post, standard adjustment methods struggle with the high dimensionality and complexity of the data.

Causal structures for confounding and mediation.

For instance, if we want to know if including a theorem ($T$) increases a paper's acceptance probability ($Y$), the "subject" of the paper ($W$) acts as a confounder: technical subjects are more likely to include theorems and may have different baseline acceptance rates. To obtain an unbiased estimate, we must "control" for the information in the text that represents these confounding factors.

Methodology: Causally Sufficient Embeddings

The authors propose a framework for learning low-dimensional representations of text that are "causally sufficient." An embedding is considered sufficient if it captures all information in the text necessary to predict both the treatment and the outcome. This is operationalized through two main architectures:

  1. Causal BERT (C-BERT): A BERT model fine-tuned with a multi-task objective. It simultaneously performs masked language modeling (unsupervised), treatment prediction, and outcome prediction.
  2. Causal Amortized Topic Model (Causal ATM): A VAE-based topic model where latent representations are regularized to reconstruct the text while remaining predictive of $T$ and $Y$.

The key innovation is the combination of supervised dimensionality reduction (targeting $T$ and $Y$) with unsupervised language modeling. The language modeling objective acts as a regularizer, ensuring the embeddings remain grounded in the semantic structure of the text and do not overfit to spurious correlations.

Experimental Setup

Evaluating causal methods is difficult because "ground truth" effects are rarely known in real-world data. The authors address this using a semi-synthetic approach across two datasets:

  • PeerRead: Scientific abstracts from arXiv where the "acceptance" outcome is simulated based on text features (e.g., presence of a theorem).
  • Reddit: Comments where the effect of a gender label on popularity is estimated, using simulated outcomes to establish a known baseline.

Empirical Performance

The proposed models, C-BERT and C-ATM, demonstrate significant improvements over traditional baselines. As shown in the table below, "causalizing" the embeddings through joint supervision and language modeling leads to estimates much closer to the ground truth than standard Bag-of-Words (BOW), unsupervised LDA, or vanilla BERT embeddings.

Comparison of causal estimation accuracy across different models and datasets.

The results highlight two critical findings:

  1. Supervision is essential: Models that do not explicitly account for the treatment and outcome in the embedding process (like standard BERT or LDA) often fail to capture the necessary confounding signal.
  2. Language Modeling improves stability: Including the unsupervised LM objective prevents the model from losing the broader context of the text, which is vital for robust causal adjustment.

Conclusion and Future Directions

This work provides a theoretically grounded and practically effective way to leverage modern NLP for causal analysis. While powerful, the "black box" nature of these deep embeddings remains a challenge for interpretability. Future research into visualization and sensitivity analysis will be crucial for making these methods more transparent. Additionally, extending this framework to handle treatments external to the text itself remains an open area for exploration.

Summary updated

Abstract

Does adding a theorem to a paper affect its chance of acceptance? Does labeling a post with the author’s gender affect the post popularity? This paper develops a method to estimate such causal effe...
BibTeX
@inproceedings{veitch2020adapting,
  author = {Victor Veitch and Dhanya Sridhar and David M. Blei},
  title = {{Adapting Text Embeddings for Causal Inference}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=rYRRUgobaG},
}
Figure