The Causal Loss: Driving Correlation to Imply Causation

Moritz Willig, Matej Zecevic, Devendra Singh Dhami, Kristian KerstingarXiv 2021 · arXiv

Abstract

Modern machine learning models excel at identifying patterns in static data but often struggle when environments change or when decision-makers intervene in a system. This paper introduces the Causal Loss, a novel, model-agnostic regularizer that encourages standard architectures—such as neural networks and decision trees—to respect underlying causal relationships. By integrating interventional data through Causal Sum-Product Networks (CaSPNs), the authors elevate classical models to "Rung 2" of Pearl’s Causal Hierarchy, enabling robust reasoning about interventions.

Introduction: Beyond Correlation

Most contemporary AI operates on the first rung of Judea Pearl’s "Ladder of Causation": Association. These models answer questions like "What if I see X?" by minimizing predictive error on observed data using standard loss functions like Cross-Entropy or Mean Squared Error. However, they often rely on spurious correlations that fail to hold under intervention—the second rung of the ladder ("What if I do X?").

A standard Directed Acyclic Graph (DAG) representing causal relationships between variables P, S, C, X, and D.

To bridge this gap, the authors propose a method to bake causal "common sense" directly into the training process. This ensures that models do not just recognize patterns but understand the mechanisms generating the data.

Methodology

The core of the proposal consists of two primary innovations designed to work with any existing model architecture.

1. The Causal Loss ($L^C$)

The Causal Loss is a regularizer added to the standard objective function. It measures the likelihood of a model's prediction against a specific interventional distribution. Instead of purely optimizing for accuracy on the training set, the model is penalized if its predictions are inconsistent with the known outcomes of system interventions. For neural networks, this is implemented as a weighted sum ($L = L^{std} + \alpha L^C$), where $\alpha$ controls the trade-off between raw correlative performance and causal consistency.

2. Causal Sum-Product Networks (CaSPNs)

Calculating the Causal Loss requires efficient estimation of interventional distributions, $P(Y|X, do(Z=z))$. The authors introduce CaSPNs—a specialized type of probabilistic circuit—to tractably estimate these complex conditional distributions.

The proposed pipeline: from causal graph and variable partitioning to the CaSPN architecture and interventional reasoning.

For Decision Trees, the authors introduce the Causal Decision Score (CDS). This replaces the traditional Gini index, guiding the tree to split nodes in a way that maximizes alignment with the causal model rather than just node purity.

Experimental Results

The approach was evaluated on synthetic benchmarks (ASIA, CANCER, EARTHQUAKE) and a new Causal Health Classification (CHC) dataset.

  • Neural Networks: On the ASIA dataset, incorporating Causal Loss improved accuracy from 82.90% to 84.72%.
  • Decision Trees: The impact was most dramatic here. Standard trees using the Gini index achieved only 44.56% accuracy on interventional tasks, while those using the Causal Loss Score jumped to 84.72%.
  • Model Compactness: Decision trees trained with Causal Loss were significantly smaller and more interpretable, effectively "finding" the true causal structure instead of overfitting to noise.

Discussion and Future Work

The Causal Loss demonstrates that specialized "causal architectures" are not always necessary; instead, causal constraints can be enforced through the objective function. This model-agnostic approach allows researchers to prioritize causal robustness by tuning the $\alpha$ hyperparameter.

However, the method currently depends on a known Structural Causal Model (SCM) to generate interventional distributions. Future research will focus on scaling these methods to larger datasets and extending the loss function to "Rung 3" of the ladder: Counterfactuals ("What would have happened if I had acted differently?").

Summary updated

Abstract

Most algorithms in classical and contemporary machine learning focus on correlation-based dependence between features to drive performance. Although success has been observed in many relevant problems, these algorithms fail when the underlying causality is inconsistent with the assumed relations. We propose a novel model-agnostic loss function called Causal Loss that improves the interventional quality of the prediction using an intervened neural-causal regularizer. In support of our theoretical results, our experimental illustration shows how causal loss bestows a non-causal associative model (like a standard neural net or decision tree) with interventional capabilities.
BibTeX
@misc{willig2021causal,
  author = {Moritz Willig and Matej Zecevic and Devendra Singh Dhami and Kristian Kersting},
  title = {{The Causal Loss: Driving Correlation to Imply Causation}},
  year = {2021},
  eprint = {2110.12066},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2110.12066},
}
Figure