TL;DR
Existing deep learning methods for causal treatment effect estimation rely on rigid, hand-crafted neural network architectures and regularizers that only suit specific causal graphs. This paper introduces NESTER, an approach that formulates causal inductive biases (such as multi-head routing and covariate balance) into a Domain-Specific Language (DSL) and uses neurosymbolic program synthesis to automatically discover tailored, differentiable causal estimation architectures.
What Problem Does This Paper Address?
Estimating causal treatment effects from observational data is notoriously difficult due to two core hurdles:
- The Fundamental Problem of Causal Inference: We only ever observe the factual outcome of the assigned treatment, never the counterfactual.
- Confounding & Selection Bias: Characteristics influencing treatment assignment also affect outcomes.
To handle these issues, popular deep learning models (such as TARNet, CFRNet, and Dragonnet) introduce specialized architectural tricks—like multi-head networks or Integral Probability Metric (IPM) regularization—to align treatment and control representations. However, these models assume fixed causal structures. For instance, IPM forces pre-treatment representations across treatment and control groups to look identical; while effective if covariates cause the treatment, this fails if treatment actually causes the covariates. Because researchers rarely know the true underlying causal graph in advance, a one-size-fits-all architecture often leads to biased estimates.
Methods
The authors reframe causal effect estimation as neurosymbolic program synthesis:
- Domain-Specific Language (DSL): Instead of fixing the neural pipeline, the authors define a grammar of differentiable operations that mirror standard causal heuristics:
if α then α else α: A smooth, sigmoid-temperature approximation of branching, equivalent to multi-head networks separating treatment and control paths.transform(α, µ, σ): Standardizes inputs to match moments across groups, mimicking IPM/Maximum Mean Discrepancy regularization.subset(α, [a..b]): Selects relevant feature slices, mimicking pre-treatment covariate selection or propensity adjustment.const(S)&⊙(α, α): Learnable constants and parameterized algebraic operators ($+, \times$).- Heuristic Search via A*: Searching through possible program structures is framed as graph search over context-free grammar rules. The authors use neural admissible relaxations, substituting unfinished non-terminal branches with overparameterized neural networks to compute an $\epsilon$-admissible heuristic $h(u)$.
- Theoretical Grounding: The authors prove that the resulting program synthesizer can approximate continuous treatment-outcome relationships up to arbitrary precision ($\epsilon$-closeness to universal approximators).
Datasets
The authors evaluate NESTER across three standard causal inference benchmarks:
- IHDP (Infant Health and Development Program): A semi-synthetic benchmark based on a randomized experiment targeting low-birth-weight, premature infants, with simulated counterfactual outcomes.
- Twins: A semi-synthetic dataset derived from twin births in the United States, evaluating mortality where one twin is treated as heavier at birth.
- Jobs (LaLonde, 1986): A real-world dataset assessing the employment and earnings impact of a national job training program, containing only factual outcomes.
No new datasets were introduced.
Key Insights and Ideas
- Inductive Biases as DSL Primitives: Instead of inventing new custom regularizers, common causal modeling practices (backdoor adjustment, representation balancing, multi-task learning) can be mapped directly to primitives in a functional grammar.
- Dynamic Architecture Discovery: NESTER does not assume a single network topology. For simple datasets like Twins, it synthesizes a lightweight feature-selection model (
subset), while for complex datasets like IHDP, it synthesizes a balanced branching pipeline (if-then-elsecombined withtransform). - Interpretability Without Sacrificing Differentiability: Because synthesized programs are compositions of human-readable operators, domain experts can inspect how covariates are routed and transformed rather than treating the model as a complete black box.
Key Results
NESTER consistently outperforms or matches state-of-the-art baselines across metrics including Average Treatment Effect error ($\epsilon_{ATE}$) and Average Treatment Effect on the Treated error ($\epsilon_{ATT}$):
- IHDP (Out-of-sample $\epsilon_{ATE}$): NESTER achieves $0.09 \pm 0.07$, substantially outperforming CFRNet/CFRWASS ($0.27 \pm 0.01$), Dragonnet ($0.29 \pm 0.31$), and multi-task Gaussian processes ($0.13 \pm 0.12$).
- Twins (Out-of-sample $\epsilon_{ATE}$): NESTER achieves $0.0063 \pm 0.0033$, outperforming deep baselines like CFRWASS ($0.0284$) and matching simple linear/matching models.
- Jobs (Out-of-sample $\epsilon_{ATT}$): NESTER achieves the lowest error at $0.02 \pm 0.01$, halving the error of Dragonnet ($0.04 \pm 0.00$) and outperforming BART ($0.08 \pm 0.03$).
Limitations and Future Work
- Search Complexity and Program Depth: To keep the search space tractable and outputs interpretable, the authors constrain synthesized programs to a maximum depth of 5. Scaling to deeper compositions or richer grammars may slow down convergence.
- In-Sample Performance on Real Data: On the Jobs benchmark, NESTER’s in-sample $\epsilon_{ATT}$ ($0.06 \pm 0.00$) lagged behind simpler baselines like OLS and GANITE ($0.01$), even though it generalized best out-of-sample.
- Future Work: The authors propose expanding the DSL with new primitives derived from unexplored causal inference heuristics and instrumental variable methods.
Summary updated