Causal Graph Modeling with Deep Neural Engines for Strong Abstract Reasoning in Language and Vision

Gaël Gendron

Abstract

Deep learning (DL) relies on discovering correlation patterns in low-level data and aggregating the information to solve a task. Despite success in a wide variety of applications, ranging from natural language to vision tasks, the learned patterns are often brittle and do not transfer out of the training data distribution (i.e. to different domains). Causality theory proposes methods to discover and estimate cause-effect relationships beyond correlations. Its powerful inference frameworks have been recently highlighted as a potential way to improve the lack of out-of-distribution generalisation in deep neural networks. However, their applications to deep learning problems remain largely under-explored. Our work attempts to bridge this gap and apply causal graphical models to abstract and causal reasoning problems in natural language and vision, requiring strong generalisation abilities beyond correlations. We integrate causal graph modelling methods into deep vision networks and Large Language Models to improve their capacity to perform strong and out-of-distribution reasoning on complex abstract problems.

Introduction: Beyond Spurious Correlations

While Deep Learning has achieved remarkable milestones, its reliance on surface-level correlations makes models "brittle." When faced with out-of-distribution (OOD) scenarios—data that deviates from the training set—these models often fail. A critical question facing the field is whether Large Language Models (LLMs) are truly reasoning or merely performing sophisticated pattern matching, a phenomenon often masked by data leakage in evaluation benchmarks.

To achieve robust intelligence, models require abstract reasoning: the capacity to identify underlying principles from minimal examples and apply them to novel contexts. This research posits that the missing link is causality—the ability to distinguish between coincidental variables and those with true functional influence.

Overview of the research framework spanning Computer Vision and Natural Language Processing.

Methodology: Integrating Causal Structures

The proposed framework bridges the gap between causality and deep learning through three distinct architectural strategies across vision and language domains.

Modular Causal LLMs (ICLM)

To move away from monolithic, dense networks, the Independent Causal Language Model (ICLM) approach decomposes the inference process into independent causal modules. A dedicated "router" selects specific modules based on input features. To prevent these modules from collapsing into a single representation, a Mutual Information (MI) loss is applied, ensuring that the learned representations remain abstract and causally independent.

Architecture of the Modular Causal LLM (ICLM) featuring independent modules and MI minimisation.

Neural-Causal GNNs for Time-Series

For dynamic tasks such as animal behavior recognition, a "Causal Structure Discovery" module is employed. This module constructs a dependency graph that informs a Graph Neural Network (GNN), ensuring that feature aggregation respects the temporal flow of cause-and-effect in the data.

Causal Variational Autoencoders (CT-VAE)

In the vision domain, reasoning is complicated by the low-level nature of pixel data. This work utilizes VAEs to "disentangle" latent variables, transforming raw pixels into high-level semantic variables. These disentangled representations serve as the foundation for subsequent causal reasoning tasks.

Experimental Evaluation and Results

The research utilized several key datasets, including a novel Abstract Reasoning Benchmark designed to test LLMs on extrapolation rather than memorization, and animal behavior data for validating causal GNNs.

Key findings include:

  • The Reasoning Gap: Current state-of-the-art LLMs struggle significantly on the new abstract reasoning benchmark, suggesting they lack fundamental causal building blocks.
  • Benefits of Modularity: The ICLM's modular design demonstrated superior OOD generalization compared to standard dense architectures. By enforcing independence, the model avoids "spurious correlations" that do not hold across different domains.
  • Structural Bias: Building causal graphs directly into the architecture acts as a powerful inductive bias, guiding the neural engine toward discovering true cause-effect relationships.

Conclusion and Future Work

While modularity and causal integration show promise, the field is in the early stages of developing models that truly "think" causally. Future research will focus on Causal Flow-of-Thoughts, regularizing LLMs to generate causal chains regardless of word order, and combining causal VAEs with Bayesian networks to enable interventional reasoning in visual scenes.

Summary updated

Abstract

Deep learning (DL) relies on discovering correlation patterns in low-level data and aggregating the information to solve a task. Despite success in a wide variety of applications, ranging from natural language to vision tasks, the learned patterns are often brittle and do not transfer out of the training data distribution (i.e. to different domains). Causality theory proposes methods to discover and estimate cause-effect relationships beyond correlations. Its powerful inference frameworks have been recently highlighted as a potential way to improve the lack of out-of-distribution generalisation in deep neural networks. However, their applications to deep learning problems remain largely under-explored. Our work attempts to bridge this gap and apply causal graphical models to abstract and causal reasoning problems in natural language and vision, requiring strong generalisation abilities beyond correlations. We integrate causal graph modelling methods into deep vision networks and Large Language Models to improve their capacity to perform strong and out-of-distribution reasoning on complex abstract problems.
BibTeX
@inproceedings{gendron2024causal,
  author = {Gaël Gendron},
  title = {{Causal Graph Modeling with Deep Neural Engines for Strong Abstract Reasoning in Language and Vision}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=yME2dT0w5E},
}
Figure