Causality is all you need

Ning Xu, Yifei Gao, Hongshuo Tian, Yongdong Zhang, Anan LiuarXiv 2023 · DOI · arXiv

Abstract

Deep learning models are often criticized for being sophisticated correlation engines that fail to account for causal relationships. This paper introduces Causal Graph Routing (CGR), a novel framework that integrates active causal intervention into neural architectures. By dynamically routing data through specialized deconfounding blocks, CGR mitigates dataset biases and outperforms larger, correlation-based models in Visual Question Answering (VQA) and Long Document Classification (LDC) with significantly fewer parameters.

Introduction: Beyond Correlation

While "correlation is not causation" is a fundamental principle, most modern AI models, including Transformers, primarily function as pattern matchers. This reliance on correlation leads to confounding, where a model learns spurious associations—such as linking "rooster crowing" with "sunrise"—without understanding the underlying mechanism. In practical applications like VQA, this manifests as a tendency to answer based on dataset frequency (e.g., predicting "sunny" for any weather query) rather than visual evidence.

Methodology: Causal Graph Routing (CGR)

The CGR framework moves beyond static architectures by treating causal inference as a dynamic routing problem. The architecture consists of stacked layers, each containing three parallel deconfounding blocks designed to address specific causal scenarios:

  1. No Confounder Block: A direct path where input predicts output via a mediator.
  2. Back-door Adjustment Block: Designed to block the influence of observable confounders, such as external common-sense knowledge.
  3. Front-door Adjustment Block: Mitigates unobservable confounders by focusing on causal flow through known mediators, such as specific visual attributes.
Causal structures used in CGR: No Confounder, Back Door, and Front Door adjustments.

Dynamic Selection via Sufficient Cause

The core innovation of CGR is the use of the Sufficient Cause concept to weight these blocks. Instead of applying a single causal adjustment to an entire task, CGR dynamically routes each sample through the most appropriate graph. This is achieved through a sharpening softmax function during training, which allows the model to explore various paths before converging on the most effective causal mechanism for a given input.

The CGR architecture showing the routing of block outputs into layer outputs across multiple causal layers.

Experimental Evaluation

The researchers evaluated CGR across three diverse benchmarks: VQA2.0 (visual question-answering), ECtHR (legal document classification), and 20 NewsGroups (topic classification).

Key Results

CGR demonstrated superior performance and efficiency:

  • VQA Accuracy: CGR achieved 75.47% on VQA2.0, surpassing the previous state-of-the-art transformer model, TRAR (72.93%).
  • NLP Performance: On the ECtHR dataset, CGR reached a 76.6 Macro F1 score, outperforming specialized models like Legal-BERT.
  • Parameter Efficiency: Remarkably, CGR outperformed ERNIE-VIL (large) using only 6 causal layers compared to ERNIE-VIL's 30-layer stack, suggesting that causal intelligence is more efficient than brute-force correlation.

Discussion and Key Insights

Qualitative analysis revealed that CGR adapts its causal strategy across layers. In VQA tasks, early layers often utilize front-door adjustments to handle visual biases, while later layers shift to back-door adjustments to integrate external knowledge. This hierarchical causal processing allows the model to build a more robust semantic understanding.

Limitations and Future Work

Currently, CGR requires pre-defined causal graphs, which necessitates domain expertise to identify mediators and confounders. Future research aims to develop pre-trained causal models that can autonomously map cause-and-effect relationships, potentially leading to AI that generalizes better to out-of-distribution scenarios.

Summary updated

Abstract

In the fundamental statistics course, students are taught to remember the well-known saying: "Correlation is not Causation". Till now, statistics (i.e., correlation) have developed various successful frameworks, such as Transformer and Pre-training large-scale models, which have stacked multiple parallel self-attention blocks to imitate a wide range of tasks. However, in the causation community, how to build an integrated causal framework still remains an untouched domain despite its excellent intervention capabilities. In this paper, we propose the Causal Graph Routing (CGR) framework, an integrated causal scheme relying entirely on the intervention mechanisms to reveal the cause-effect forces hidden in data. Specifically, CGR is composed of a stack of causal layers. Each layer includes a set of parallel deconfounding blocks from different causal graphs. We combine these blocks via the concept of the proposed sufficient cause, which allows the model to dynamically select the suitable deconfounding methods in each layer. CGR is implemented as the stacked networks, integrating no confounder, back-door adjustment, front-door adjustment, and probability of sufficient cause. We evaluate this framework on two classical tasks of CV and NLP. Experiments show CGR can surpass the current state-of-the-art methods on both Visual Question Answer and Long Document Classification tasks. In particular, CGR has great potential in building the "causal" pre-training large-scale model that effectively generalizes to diverse tasks. It will improve the machines' comprehension of causal relationships within a broader semantic space.
BibTeX
@misc{xu2023causality,
  author = {Ning Xu and Yifei Gao and Hongshuo Tian and Yongdong Zhang and Anan Liu},
  title = {{Causality is all you need}},
  year = {2023},
  eprint = {2311.12307},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2311.12307},
}
Figure