Composing Modeling and Simulation with Machine Learning in Julia

Christopher Rackauckas, Ranjan Anantharaman, Alan Edelman, Shashi Gowda, Maja Gwózdz, Anand Jain, Chris Laughman, Yingbo Ma, Francesco Martinuzzi, Avik Pal, Utkarsh Rajput, Elliot Saba, Viral B. Shah

TL;DR

This paper introduces JuliaSim, a new environment that seamlessly integrates traditional modeling and simulation with machine learning. It achieves this by building accelerated "surrogate" models using Continuous-Time Echo State Networks (CTESNs) within an acausal modeling framework (ModelingToolkit.jl), enabling significant speedups for complex system simulations and design optimization.

What Problem Does This Paper Address?

Modern engineering simulations, especially those involving detailed multi-physics component models, are computationally expensive. This high cost makes design, optimization, and control tasks prohibitively time-consuming. While machine learning (ML) offers a promising avenue for creating reduced-order models or "surrogates" to speed up simulations, current methods often lack robustness, making it difficult for modelers to trust their accuracy or integrate them seamlessly into existing workflows. The core challenge is to bridge the gap between the rigorous, symbolic nature of traditional modeling and the data-driven, often approximate nature of ML, creating a unified environment that leverages the strengths of both.

Methods

The paper introduces JuliaSim, built upon ModelingToolkit.jl (MTK), an acausal modeling language for Julia. MTK allows for the hierarchical composition of models and supports a "staged compilation" process that can incorporate both exact and inexact (ML-based) transformations.

The key ML technique employed is the Continuous-Time Echo State Network (CTESN). CTESNs are presented as a way to represent trained ML models as differential-algebraic equations (DAEs), making them compatible with traditional simulation frameworks. CTESNs are chosen for their ability to handle "stiff" systems (systems with widely varying time scales), a common challenge in engineering simulations, by using an implicit training process that avoids local optimization algorithms prone to instability.

JuliaSim also leverages the Functional Mock-up Interface (FMI) standard, allowing for the integration and co-simulation of models developed in different tools. This enables engineers to replace components within existing simulation workflows with their accelerated surrogate counterparts.

Datasets

The paper uses the Chicago O'Hare TMY3 weather dataset to define ambient conditions for the HVAC system simulation.

For evaluation, they use a detailed Room Air Conditioner (RAC) model exported as a Functional Mock-up Unit (FMU). This model is used to generate training data by simulating it with 100 parameter sets sampled using Latin hypercube sampling. The surrogate's performance is then evaluated on a separate set of test parameters.

They also use the Robertson equations, a canonical stiff benchmark problem, to demonstrate the effectiveness of CTESNs on challenging systems.

Key Insights and Ideas

  • Bridging the gap: The core innovation is the seamless integration of ML surrogates into a robust, acausal modeling framework. This moves beyond simply using ML for prediction to embedding it as a composable component within traditional simulation workflows.
  • CTESNs for stiff systems: CTESNs offer a novel approach to creating ML surrogates that are inherently DAEs, making them compatible with existing simulation infrastructure and capable of handling the stiff dynamics common in physical systems.
  • Composability is key: The acausal nature of MTK and the adoption of FMI are crucial for enabling the "drop-in" replacement of complex components with their faster surrogate versions, facilitating exploration of design spaces and co-simulation.
  • Automated surrogate generation: JuliaSim provides a pipeline for automating the creation and deployment of these ML surrogates, reducing the manual effort required for modelers.

Key Results

  • HVAC Surrogate Performance: A CTESN surrogate for an HVAC system achieved less than 4% relative error while accelerating simulation by 340x.
  • Global Optimization Speedup: Using the surrogate for global parameter optimization of the HVAC system yielded a two orders of magnitude speedup compared to using the full model.
  • Robertson Equations: Surrogates for the stiff Robertson equations showed significantly reduced reservoir sizes (e.g., 3 vs. 3000) with low relative errors (e.g., 0.0200% for NPCTESN).
  • Co-simulation Integration: The surrogate was successfully deployed in a co-simulation loop, acting as a drop-in replacement for an FMU, demonstrating its practical utility in complex system simulations.

Limitations and Future Work

  • Surrogate Robustness: While CTESNs are more robust than some other ML approaches, the paper notes that the surrogate's prediction can smooth over transients, and error can peak at points with discontinuous derivatives. Increasing the number of training samples or reducing the parameter space range can mitigate this.
  • FMI Co-simulation: The paper mentions that the co-simulation example uses simplified models. Future work includes demonstrating this for larger coupled simulations with more FMUs or more complex models.
  • GUI for Surrogatization: A GUI for users to surrogatize their own components is mentioned as a planned feature.
  • Additional Surrogate Algorithms: The authors plan to add other surrogate algorithms like proper orthogonal decomposition, neural ODEs, and dynamic mode decomposition in future releases.
  • Symbolic Simplification: Exploring the integration of ML within symbolic simplification algorithms is also a future direction.
  • Model Exchange FMUs: Integration with state-of-the-art solvers for simulating Model-Exchange FMUs is planned for future releases.

Summary updated

Abstract

In this paper we introduce JuliaSim, a high-performance programming environment designed to blend traditional modeling and simulation with machine learning. JuliaSim can build accelerated surrogates from component-based models, such as those conforming to the FMI standard, using continuous-time echo state networks (CTESN). The foundation of this environment, ModelingToolkit.jl, is an acausal modeling language which can compose the trained surrogates as components within its staged compilation process. As a complementary factor we present the JuliaSim model library, a standard library with differential-algebraic equations and pre-trained surrogates, which can be composed using the modeling system for design, optimization, and control. We demonstrate the effectiveness of the surrogate-accelerated modeling and simulation approach on HVAC dynamics by showing that the CTESN surrogates accurately capture the dynamics of a HVAC cycle at less than 4\% error while accelerating its simulation by 340x. We illustrate the use of surrogate acceleration in the design process via global optimization of simulation parameters using the embedded surrogate, yielding a speedup of two orders of magnitude to find the optimum. We showcase the surrogate deployed in a co-simulation loop, as a drop-in replacement for one of the coupled FMUs, allowing engineers to effectively explore the design space of a coupled system. Together this demonstrates a workflow for automating the integration of machine learning techniques into traditional modeling and simulation processes.
BibTeX
@inproceedings{rackauckas2021composing,
  author = {Christopher Rackauckas and Ranjan Anantharaman and Alan Edelman and Shashi Gowda and Maja Gwózdz and Anand Jain and Chris Laughman and Yingbo Ma and Francesco Martinuzzi and Avik Pal and Utkarsh Rajput and Elliot Saba and Viral B. Shah},
  title = {{Composing Modeling and Simulation with Machine Learning in Julia}},
  booktitle = {Linköping},
  year = {2021},
  doi = {10.3384/ecp2118197},
}
Figure