Causal disentanglement of multimodal data

Elise Walker, Jonas A. Actor, Carianne Martinez, Nathaniel Trask

Abstract

Understanding causal relationships in scientific data is a fundamental challenge, often hindered by the lack of labeled interventional data. We present causalPIMA, a framework for unsupervised causal discovery from multimodal data. By integrating variational autoencoders (VAEs) with differentiable directed acyclic graph (DAG) learning and physics-informed decoders, causalPIMA identifies causal structures without manual intervention. Evaluated on synthetic and real-world 3D-printed lattice datasets, the model successfully disentangles causal factors and maps their influence on physical responses.

Introduction

In scientific discovery, distinguishing between correlation and causation is critical. While deep learning excels at finding correlations, identifying how one feature causes another typically requires interventional data or prior structural knowledge—luxuries often unavailable in exploratory research. Traditional disentangled representation learning aims to separate independent factors but frequently overlooks the causal dependencies between them.

To address this, we introduce causalPIMA (Causal Physics-Informed Multimodal Autoencoder). This framework leverages the richness of multimodal data (e.g., images and sensor readings) and physical constraints to discover causal structures in a completely unsupervised manner.

Methods

The causalPIMA architecture is built upon three core components designed to bridge the gap between raw data and causal logic.

Architecture of the causalPIMA framework, integrating multimodal encoders, a causal GMM latent space, and physics-informed decoders.

1. Multimodal Variational Autoencoder

The model employs separate encoders for different data modalities, such as images and time-series curves. These representations are fused into a shared latent space using a "Product of Experts" approach, ensuring that the latent representation captures information consistent across all views of the data.

2. Differentiable DAG Learning

Discovering the "best" causal graph is traditionally a combinatorial problem. causalPIMA utilizes a continuous optimization strategy inspired by Hodge theory. By assigning scores to nodes and ensuring that the "flow" between them follows these scores, the model mathematically guarantees the resulting structure is a Directed Acyclic Graph (DAG), avoiding infinite causal loops. This allows causal discovery to be integrated directly into the gradient-based training process.

3. Causal GMM Prior

Unlike standard VAEs that assume a simple Gaussian prior, causalPIMA uses a Gaussian Mixture Model (GMM). Each cluster in the mixture corresponds to a specific outcome or "phenotype" within the discovered causal graph. This links the geometric structure of the latent space directly to the underlying causal logic.

Furthermore, the framework incorporates physics-informed decoders. By embedding known physical laws—such as the mechanics of a stress-strain curve—into the decoder, the model is constrained to find latent features that are physically meaningful, clarifying the causal relationships.

Experimental Evaluation

The model was tested on two distinct datasets to evaluate its ability to recover hidden causal structures.

Synthetic Circles

A dataset of 4,096 images was generated where circles varied by hue, radius, and shift. The data generation process included a hidden causal structure where radius and hue influenced the probability of the shift. causalPIMA successfully identified "Radius" and "Hue" as the parent nodes and correctly clustered the data into eight distinct phenotypes based on these factors.

3D Printed Lattices

In a real-world application, the model analyzed 91 samples of 3D-printed lattices. Each sample consisted of an image (geometry) and a stress-strain curve (mechanical response). The model discovered that the lattice type (e.g., octet vs. gyroid) acted as the causal factor influencing the resulting stress-strain profile. It successfully separated these geometries in the latent space and mapped how they led to specific mechanical behaviors.

Discussion and Key Insights

A central finding of this work is that multimodality can serve as a form of supervision. Having multiple "views" of the same phenomenon provides sufficient internal structure to substitute for human labels. Additionally, the "curl-free" trick for DAG learning transforms a hard combinatorial problem into one solvable with standard deep learning toolkits.

By using an expert physics model as an anchor in the decoding process, the system is forced to align its latent representations with physical reality. This end-to-end training approach—optimizing encoders, decoders, cluster centers, and the causal graph simultaneously—proves robust for discovering plausible causal structures.

Limitations and Future Work

While causalPIMA shows promise for exploratory science, it has limitations. In a fully unsupervised setting, the model may find a "plausible" graph rather than a unique, true causal structure. Currently, the framework focuses on discrete categorical nodes; extending this to continuous causal variables remains a significant challenge. Additionally, as the number of causal factors increases, the complexity of the joint distribution grows, necessitating future research into more scalable architectures.

Summary updated

BibTeX
@misc{walker2023causal,
  author = {Elise Walker and Jonas A. Actor and Carianne Martinez and Nathaniel Trask},
  title = {{Causal disentanglement of multimodal data}},
  year = {2023},
  eprint = {2310.18471},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.18471},
}
Figure