Unsupervised physics-informed disentanglement of multimodal data for high-throughput scientific discovery

Nathaniel Trask, Carianne Martinez, Kookjin Lee, Brad BoycearXiv 2022 · arXiv

Introduction

Modern automated laboratories generate massive, heterogeneous datasets at an unprecedented rate. Making sense of this multimodal data—ranging from high-resolution images to complex sensor time-series—is a critical bottleneck in scientific discovery. This paper introduces PIMA (Physics-Informed Multimodal Autoencoders), a variational inference framework designed to fuse disparate data types into a shared, physically structured latent space without the need for manual labels.

The Challenge of Multimodal Scientific Data

In fields such as materials science, researchers seek "fingerprints"—easily measurable signals (like a photograph) that correlate with difficult-to-measure physical properties (like mechanical stress). However, two major obstacles hinder progress:

  1. Labeling Scarcity: The sheer volume of data produced by high-throughput experiments makes manual labeling impractical.
  2. Lack of Physical Context: Conventional machine learning models often ignore the underlying physical laws, leading to representations that may be statistically valid but physically nonsensical.

The PIMA Framework

The PIMA architecture addresses these challenges by integrating physical constraints directly into the learning process.

Overview of the PIMA architecture, showing the fusion of multimodal encoders and the physics-informed decoder.

The framework consists of several key components:

  • Product of Experts (PoE) Fusion: This mechanism encodes individual data modalities separately and then merges them into a single joint representation in the latent space.
  • Gaussian Mixture Model (GMM) Prior: Instead of a standard normal distribution, PIMA utilizes a GMM to automatically organize data into unsupervised clusters, representing different physical regimes or material categories.
  • Physics-Informed Mixture of Experts (MoE) Decoder: This component incorporates actual physical models (e.g., strain-hardening equations) into the decoding process. This "physics-based inductive bias" ensures that the learned representations remain consistent with natural laws.
  • Cross-modal Inference: Once trained, the model can perform inference across modalities, such as predicting a mechanical stress-strain curve from a simple image.

Experimental Evaluation

The researchers validated PIMA using two distinct datasets:

  1. Multimodal MNIST: A synthetic benchmark pairing digit images with 1D functions where the slope is mathematically linked to the digit's identity.
  2. Lattice Metamaterials: A real-world dataset comprising SEM images of 3D-printed metal lattices (gyroid and octet) paired with their mechanical stress-strain curves.
Lattice metamaterials dataset showing SEM images of 3D-printed structures and their corresponding stress-strain curves.

Results and Discussion

PIMA demonstrated significant improvements over existing methods:

  • Accuracy and Robustness: On the Multimodal MNIST benchmark, PIMA achieved 100% unsupervised clustering accuracy. It also showed remarkable resilience to noise, maintaining over 98% accuracy even when secondary modalities were heavily corrupted.
  • Scientific Discovery: On the lattice metamaterials data, PIMA achieved 94.74% accuracy in lattice identification and successfully reconstructed stress-strain curves from images alone. This demonstrates the model's ability to automate "fingerprinting" tasks that typically require expert intuition.
  • Latent Space Organization: The model did not just cluster data; it ordered the latent space according to physical parameters (like slope in the MNIST test), suggesting an ability to discover continuous physical transitions.

Conclusion

PIMA represents a significant step toward unsupervised, physics-aware machine learning for scientific discovery. By embedding physical laws as regularizers, the framework achieves superior disentanglement and cross-modal predictive power. Future work will focus on incorporating more complex physical models, such as those based on partial differential equations, and optimizing the framework for streaming, large-scale datasets.

Summary updated

Abstract

We introduce physics-informed multimodal autoencoders (PIMA) - a variational inference framework for discovering shared information in multimodal scientific datasets representative of high-throughput testing. Individual modalities are embedded into a shared latent space and fused through a product of experts formulation, enabling a Gaussian mixture prior to identify shared features. Sampling from clusters allows cross-modal generative modeling, with a mixture of expert decoder imposing inductive biases encoding prior scientific knowledge and imparting structured disentanglement of the latent space. This approach enables discovery of fingerprints which may be detected in high-dimensional heterogeneous datasets, avoiding traditional bottlenecks related to high-fidelity measurement and characterization. Motivated by accelerated co-design and optimization of materials manufacturing processes, a dataset of lattice metamaterials from metal additive manufacturing demonstrates accurate cross modal inference between images of mesoscale topology and mechanical stress-strain response.
BibTeX
@misc{trask2022unsupervised,
  author = {Nathaniel Trask and Carianne Martinez and Kookjin Lee and Brad Boyce},
  title = {{Unsupervised physics-informed disentanglement of multimodal data for high-throughput scientific discovery}},
  year = {2022},
  eprint = {2202.03242},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2202.03242},
}
Figure