Abstract
Deep learning (DL) relies on discovering correlation patterns in low-level data and aggregating the information to solve a task. Despite success in a wide variety of applications, ranging from natural language to vision tasks, the learned patterns are often brittle and do not transfer out of the training data distribution (i.e. to different domains). Causality theory proposes methods to discover and estimate cause-effect relationships beyond correlations. Its powerful inference frameworks have been recently highlighted as a potential way to improve the lack of out-of-distribution generalisation in deep neural networks. However, their applications to deep learning problems remain largely under-explored. Our work attempts to bridge this gap and apply causal graphical models to abstract and causal reasoning problems in natural language and vision, requiring strong generalisation abilities beyond correlations. We integrate causal graph modelling methods into deep vision networks and Large Language Models to improve their capacity to perform strong and out-of-distribution reasoning on complex abstract problems.
Introduction: Beyond Spurious Correlations
While Deep Learning has achieved remarkable milestones, its reliance on surface-level correlations makes models "brittle." When faced with out-of-distribution (OOD) scenarios—data that deviates from the training set—these models often fail. A critical question facing the field is whether Large Language Models (LLMs) are truly reasoning or merely performing sophisticated pattern matching, a phenomenon often masked by data leakage in evaluation benchmarks.
To achieve robust intelligence, models require abstract reasoning: the capacity to identify underlying principles from minimal examples and apply them to novel contexts. This research posits that the missing link is causality—the ability to distinguish between coincidental variables and those with true functional influence.
Methodology: Integrating Causal Structures
The proposed framework bridges the gap between causality and deep learning through three distinct architectural strategies across vision and language domains.
Modular Causal LLMs (ICLM)
To move away from monolithic, dense networks, the Independent Causal Language Model (ICLM) approach decomposes the inference process into independent causal modules. A dedicated "router" selects specific modules based on input features. To prevent these modules from collapsing into a single representation, a Mutual Information (MI) loss is applied, ensuring that the learned representations remain abstract and causally independent.
Neural-Causal GNNs for Time-Series
For dynamic tasks such as animal behavior recognition, a "Causal Structure Discovery" module is employed. This module constructs a dependency graph that informs a Graph Neural Network (GNN), ensuring that feature aggregation respects the temporal flow of cause-and-effect in the data.
Causal Variational Autoencoders (CT-VAE)
In the vision domain, reasoning is complicated by the low-level nature of pixel data. This work utilizes VAEs to "disentangle" latent variables, transforming raw pixels into high-level semantic variables. These disentangled representations serve as the foundation for subsequent causal reasoning tasks.
Experimental Evaluation and Results
The research utilized several key datasets, including a novel Abstract Reasoning Benchmark designed to test LLMs on extrapolation rather than memorization, and animal behavior data for validating causal GNNs.
Key findings include:
- The Reasoning Gap: Current state-of-the-art LLMs struggle significantly on the new abstract reasoning benchmark, suggesting they lack fundamental causal building blocks.
- Benefits of Modularity: The ICLM's modular design demonstrated superior OOD generalization compared to standard dense architectures. By enforcing independence, the model avoids "spurious correlations" that do not hold across different domains.
- Structural Bias: Building causal graphs directly into the architecture acts as a powerful inductive bias, guiding the neural engine toward discovering true cause-effect relationships.
Conclusion and Future Work
While modularity and causal integration show promise, the field is in the early stages of developing models that truly "think" causally. Future research will focus on Causal Flow-of-Thoughts, regularizing LLMs to generate causal chains regardless of word order, and combining causal VAEs with Bayesian networks to enable interventional reasoning in visual scenes.
Summary updated