Causal Inference and Discovery with Latent Confounders

This direction is merged into another direction: unobserved confounding (unmeasured confounding). Its last version stays readable and citable here.

Version 1 of 1170 papers177 conceptsfrozen 2026-10-04

In the ideal world of causal inference, we observe every variable that matters. If we want to know how a drug dosage affects a patient’s recovery, we simply control for their age, weight, and pre-existing conditions. But in the real world, the most influential factors are often invisible. A patient’s genetic predisposition, a student’s internal motivation, or a defendant’s courtroom demeanor are rarely captured in spreadsheets.

These unobserved confounders—hidden variables that influence both the "treatment" (the cause) and the "outcome" (the effect)—create a fundamental barrier. They introduce spurious correlations that lead standard machine learning models to mistake a coincidence for a mechanism. Recent research has shifted from assuming "full observability" to developing frameworks that explicitly model these latent drivers.

The Latent Variable as a Surrogate

If we cannot measure a confounder directly, can we infer its presence from the patterns it leaves behind? This is the intuition behind using generative models like Variational Autoencoders (VAEs) for causal tasks.

Yang Li and colleagues introduced ContiVAE (Estimating Individual Dose-Response Curves under Unobserved Confounders from Observational Data), which uses a "Tilted Gaussian" prior to capture individual differences. By modeling unobserved confounders as latent variables, the framework maintains predictive stability even when treatment assignment is highly non-random. Similarly, Matthew James Vowels, Richard Bowden, and Necati Cihan Camgöz developed Targeted VAE (Targeted VAE: Structured Inference and Targeted Learning for Causal Parameter Estimation). This model structures its latent space to specifically "disentangle" confounders from other noise, using a targeted regularizer to ensure the latent representation is optimized for causal estimation rather than just data reconstruction.

The core idea is that the observed features are "proxies" for the hidden truth. If a VAE can compress the observed data into a meaningful latent space, that space likely contains the "ghost" of the unobserved confounder.

Finding Levers with Instrumental Variables

When we can't see the confounder, we often look for an Instrumental Variable (IV)—a "lever" that affects the treatment but has no direct connection to the outcome except through that treatment.

In many datasets, however, a perfect instrument doesn't exist. Debo Cheng, Jiuyong Li, and their team proposed CIV.VAE (Causal Inference with Conditional Instruments using Deep Generative Models), which uses deep learning to discover "Conditional Instrumental Variables" (CIVs) directly from raw data. This allows researchers to calculate unbiased effects even when the "perfect" instrument is hidden.

In survival analysis, Andrew Ying, Ronghui Xu, and James D. Murphy applied a similar logic using Two-Stage Residual Inclusion (2SRI) (Two-Stage Residual Inclusion under the Additive Hazards Model). By modeling the "residuals" (the part of the treatment that the instrument can't explain), they can effectively isolate and subtract the bias introduced by unmeasured factors in complex medical data.

Discovering the Hidden Graph

Beyond estimating the strength of a relationship, researchers are now trying to map the structure of causal networks when some nodes are missing.

Traditional causal discovery algorithms often fail if they assume all nodes are observed. Haoyue Dai, Yiwen Qiu, and Kun Zhang addressed this by introducing generalized rank constraints (Latent Variable Causal Discovery under Selection Bias). Instead of relying on simple correlations, they look at the algebraic rank of covariance matrices. This mathematical signature allows them to detect the presence of hidden variables and selection bias simultaneously, recovering the "skeleton" of the causal graph even from systematically biased data.

This structural perspective is even being applied to large-scale models. Raanan Y. Rohekar and colleagues (Causal Interpretation of Self-Attention in Pre-Trained Transformers) found that the self-attention mechanism in Transformers can be viewed as an implicit Structural Equation Model (SEM). Their method, CLEANN, attempts to extract these causal structures to explain model predictions, though they note that latent confounders in text (like author intent) still pose a significant challenge for producing human-readable explanations.

Invariance as a Shield Against Bias

Why does this matter for general machine learning? Because models that ignore latent confounders are fragile. They rely on spurious correlations—patterns that exist in the training data but disappear when the environment changes.

Paras Sheth, huan liu, and others argue that Domain Generalization (DG) is essentially a causal problem (Domain Generalization - A Causal Perspective). If a model learns the true causal mechanism, that mechanism should be "invariant" across different domains. However, if unobserved variables influence both the input and the target, the model may latch onto a shortcut that only works in the source domain.

By using techniques like causal backdoor adjustment, researchers like Siyin Wang and Jie Zhou (Domain Generalization via Causal Adjustment for Cross-Domain Sentiment Analysis) are building models that "de-bias" themselves. They disentangle domain-invariant causal features from domain-specific noise, allowing models to maintain accuracy even when moved to entirely new environments where the hidden "rules" of the data have shifted.

Papers in this version

Showing 60 of 170 papers.

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_causal_inference_and_discovery_with_late_db3791f2,
  title = {Causal Inference and Discovery with Latent Confounders},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/causal-inference-and-discovery-with-latent-confounders/rdr_db3791f2de4a4755a3d33a36ef5fe1b8},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_causal_inference_and_discovery_with_late_db3791f2_v1,
  title = {Causal Inference and Discovery with Latent Confounders},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/causal-inference-and-discovery-with-latent-confounders/rdr_db3791f2de4a4755a3d33a36ef5fe1b8/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}