datafold: data-driven models for point clouds and time series on manifolds

Daniel Lehmberg, Felix Dietrich, Gerta Köster, Hans‐Joachim BungartzJ. Open Source Softw 2020 · DOI · DBLP

Introduction

In the era of high-resolution sensors and large-scale simulations, researchers are often overwhelmed by high-dimensional datasets. Despite the apparent complexity, such data frequently resides on a lower-dimensional structure known as a manifold. datafold is a Python library designed to bridge the gap between manifold learning and dynamical systems, providing a unified framework to discover these hidden structures and model the underlying rules governing temporal evolution.

The Manifold Hypothesis and Geometric Discovery

The manifold hypothesis suggests that high-dimensional data—whether it be pixel values in an image or state variables in a simulation—actually sits on a lower-dimensional "surface" within the high-dimensional space. Identifying this surface is crucial for reducing complexity and gaining physical insight. datafold addresses this by providing tools to parameterize these manifolds and model dynamics directly on them, bypassing the need for explicit governing equations.

Core Methodology

Manifold Learning with Diffusion Maps

The primary tool for geometric analysis in datafold is the Diffusion Map. This algorithm uses a kernel function to measure proximity between data points, constructing a global coordinate system that preserves the essential geometric structure of the dataset. This approach is particularly effective for non-linear dimensionality reduction, as demonstrated with datasets like handwritten digits.

A 2D embedding of the handwritten digits dataset (0-5) using Diffusion Maps, showing how the library preserves geometric relationships between high-dimensional image vectors.

Modeling Dynamics with DMD and E-DMD

For time-series data, datafold implements Dynamic Mode Decomposition (DMD) and Extended DMD (E-DMD). These methods decompose non-linear dynamics into linear spatio-temporal components. By "lifting" data into a higher-dimensional space using a dictionary of functions, E-DMD approximates the Koopman operator, allowing for the linearization and prediction of complex systems without needing the original physics equations.

A 3D visualization of a system's phase space, illustrating how datafold captures the geometric structure of trajectories over time.

Software Architecture and Integration

A key strength of datafold is its scikit-learn-compatible API. By adopting the standard fit/transform/predict workflow, it integrates seamlessly into existing machine learning pipelines. The library is designed with a clear hierarchy, from low-level data structures to high-level "meta-models," allowing researchers to swap kernels or algorithms with minimal friction. This modularity makes it an ideal "glue" layer between specialized libraries like PyDMD and PySINDy.

Key Features and Performance

  • Equation-Free Surrogates: datafold enables the creation of fast, data-driven surrogate models for complex systems, such as those governed by heavy partial differential equations (PDEs), using only simulation or experimental data.
  • Out-of-Sample Extension: Using methods like Geometric Harmonics, the library can efficiently interpolate values for millions of new data points in seconds, overcoming a common bottleneck in manifold learning.
  • Scalability: The implementation utilizes sparse kernel matrix representations to ensure the software scales effectively to large, real-world datasets.

Conclusion

datafold provides a robust, modular environment for researchers in fields ranging from fluid dynamics to chemical kinetics. While challenges remain in automated model selection—such as choosing the optimal dictionary for E-DMD—the library significantly lowers the barrier to applying advanced manifold learning and Koopman operator theory to complex, high-dimensional data.

Summary updated

BibTeX
@article{lehmberg2020datafold,
  author = {Daniel Lehmberg and Felix Dietrich and Gerta Köster and Hans‐Joachim Bungartz},
  title = {{datafold: data-driven models for point clouds and time series on manifolds}},
  journal = {J. Open Source Softw.},
  year = {2020},
  doi = {10.21105/joss.02283},
}
Figure