To predict how an individual will respond to a specific intervention—whether a medical treatment, a marketing campaign, or a credit policy—researchers must estimate Individual Treatment Effects (ITE). In observational data, this is notoriously difficult because of selection bias: the reasons why someone received a treatment are often correlated with their eventual outcome.
To address this, a dominant strategy in deep learning has been to learn "balanced" representations. The goal is to transform high-dimensional data into a latent space where the treated and control groups are indistinguishable, effectively turning an observational study into a synthetic randomized controlled trial. However, a growing body of research reveals a fundamental conflict: the very process of removing bias can inadvertently strip away the predictive information necessary to forecast outcomes.
The Balancing Act
The foundation of modern causal representation learning rests on the idea of domain adaptation. As Johansson et al. and Shalit et al. proposed, we can minimize the distance between the distributions of treated and control groups in a latent space using regularization techniques like Integral Probability Metrics (IPM) or Maximum Mean Discrepancy (MMD).
By forcing the model to ignore features that distinguish the two groups, we theoretically mitigate the confounding bias that plagues observational data. This allows the model to generalize from the observed group to the counterfactual group (the "what if" scenario).
The Over-balancing Trap
While balancing reduces bias, it introduces a new risk: over-balancing. Bellot et al. argue that forcing treated and control distributions to be identical can "throw the baby out with the bathwater." If a feature is predictive of both the treatment assignment and the outcome (a confounder), or even just the outcome (an adjustment variable), aggressive balancing may erase it from the representation.
This creates a fundamental trade-off between distributional balance and predictive sufficiency. As Atan et al. and Huang et al. describe, if the regularization weight for balancing is set too high, the model may learn a trivial, constant representation that is perfectly unbiased but entirely useless for distinguishing which individuals will actually benefit from the treatment.
Differentiating Feature Roles
The intuition behind avoiding over-balancing lies in recognizing that not all variables should be treated equally. Liu et al. and Molaei et al. suggest that features generally fall into three categories:
- Confounders: Variables that influence both treatment and outcome. These must be balanced but preserved.
- Instrumental Variables: Variables that influence treatment but not the outcome. Balancing these is unnecessary and can actually amplify bias.
- Adjustment (Prognostic) Variables: Variables that influence the outcome but not the treatment. Balancing these is harmful because it removes pure predictive signal without any bias-reduction benefit.
When models like SCRNet (Liu et al.) explicitly separate these features, they can maintain the "raw" state of adjustment variables while only debiasing true confounders. This ensures the model retains the nuanced signals required for accurate counterfactual estimation.
Toward Sufficient Representations
Recent work has shifted toward "sufficient representation learning," which seeks a latent space that satisfies two conditions simultaneously: it must be independent of treatment assignment (to ensure causal validity) while remaining highly predictive of potential outcomes.
Several mechanisms have emerged to navigate this trade-off:
- Information-Theoretic Regularization: Instead of unstable adversarial training, Tang proposes using a direct penalty on mutual information. This allows for a tunable hyperparameter that controls the exact balance between removing assignment bias and maintaining predictive capacity.
- Invertible Representations: To prevent the loss of historical data in time-series settings, El Bouchattaoui et al. emphasize that representations should be invertible. This ensures that the compressed state still contains all necessary information to reconstruct the original input, preventing the accidental deletion of confounding factors.
- Double Balancing: For complex scenarios like continuous dose-response curves, Zhu et al. introduced "double balancing" frameworks. These ensure that features are made independent of treatment levels while using contrastive learning to preserve the prognostic information that defines how different individuals respond to different doses.
Ultimately, the goal is to move beyond simple group similarity. As Yan et al. suggest with their G-learner framework, we might even avoid transforming the data altogether, instead adjusting the prediction model itself along a geometric path (a Wasserstein geodesic) to bridge the gap between treatment groups without ever discarding the underlying feature diversity.
Key Foundational Work
Estimating individual treatment effect: generalization bounds and algorithms (2017) — contributed 1 concept elements to this direction.
Learning Representations for Counterfactual Inference (2016) — contributed 1 concept elements to this direction.
Learning Representations for Counterfactual Inference (2016) — contributed 1 concept elements to this direction.