An Invariant Compiler for Neural ODEs in AI-Accelerated Scientific Simulation

Fangzhou Yu, Yiqi Su, Ray Lee, Shenfeng Cheng, Naren RamakrishnanarXiv 2026 · arXiv

TL;DR

Neural Ordinary Differential Equations (Neural ODEs) are powerful tools for modeling continuous-time systems, but they frequently drift over long horizons and violate fundamental physical laws like the conservation of energy or mass. This paper introduces the Invariant Compiler, an LLM-driven framework that automatically translates high-level physical specifications into structure-preserving Neural ODE architectures. By treating physical invariants as first-class types, the framework guarantees that the model's trajectories strictly adhere to domain constraints by construction.

What Problem Does This Paper Address?

Data-driven models like Neural ODEs (NODEs) excel at learning complex dynamics directly from data by parameterizing a system's vector field with a neural network. However, standard NODEs are "physics-agnostic." When forecasting over long time horizons, their predictions tend to drift away from the true underlying physical manifold, leading to nonsensical results—such as creating mass out of nothing or violating the conservation of momentum.

Existing attempts to fix this fall into two camps, both with major drawbacks:

  1. Soft Penalties: Adding regularization terms to the loss function encourages the model to respect constraints, but offers no formal guarantees. The model can still drift, and multiple conflicting penalties often disrupt the optimization process.
  2. Bespoke Architectures: Researchers design specialized neural networks (like Hamiltonian or Lagrangian Neural Networks) tailored to specific constraints. While exact, this approach is highly fragmented; a new architecture must be manually derived and coded for every new type of physical invariant.

The Invariant Compiler bridges this gap by providing a unified, automated pipeline that treats physical invariants as structural type signatures rather than special cases.

Methods

The core philosophy of the Invariant Compiler is the strict separation of what must be preserved (scientific structure) from what must be learned (dynamics within that structure).

As illustrated in the pipeline overview, the workflow consists of six distinct stages:

The Invariant Compiler Pipeline overview, demonstrating the workflow from input ODEs to simulation using an epidemiological SIR model as an example.
  1. Input ODEs: The user provides a generic, unconstrained symbolic or executable definition of the system dynamics.
  2. Invariant Typing: The system identifies constraints either via user specification (e.g., conservation laws) or data-driven discovery (e.g., symbolic regression).
  3. Prompt (Program Specification): The base dynamics and typed invariants are structured into a precise prompt for a Large Language Model (LLM).
  4. Invariant Compiler: Acting as a deterministic code-rewrite engine, the LLM maps the invariant types to a Geometric Intermediate Representation (IR) (e.g., mapping conservation laws to null spaces or simplices to spheres) and constructs a structure-preserving vector field tangent to the constraint manifold.
  5. Generate Neural ODE Code: The compiler outputs executable PyTorch code (nn.Module) where the invariant-preserving geometry is fixed, leaving only the unconstrained submodules learnable.
  6. Simulation & Learning: The compiled network is trained and simulated using standard numerical ODE solvers.

The authors cataloged and implemented rewrite rules for a wide array of invariant types, including Simplex/Norm preservation (via skew-symmetric dynamics), Lorentz and PSD Cones (via tangent-space projection and Cholesky parameterization), Stoichiometric Constraints (via null-space projection), and Hamiltonian/Poisson/GENERIC thermodynamic structures.

Datasets

The framework was evaluated across 11 synthetic dynamical systems spanning physics, chemistry, biology, and epidemiology. These benchmarks include:

  • SIR & Replicator-Mutator models (Simplex constraints)
  • NOx & Six-Species Chemical Reaction Networks (Stoichiometric constraints)
  • Lorentz Cone Spiral & Coupled Radial-Angular Dynamics (Cone constraints)
  • Lotka-Volterra & Damped Harmonic Oscillators (Hamiltonian/Port-Hamiltonian structures)
  • Thermomechanical Systems (GENERIC thermodynamic structure)
  • Extended Pendulum & Two-Body Gravitational Systems (Multi-invariant/Poisson structures)

Key Insights and Ideas

  • LLMs as Compilers, Not Predictors: Instead of using an LLM to predict physical states (which is notoriously unreliable), this work uses the LLM as a zero-temperature code generation and synthesis engine. It translates high-level physical properties into exact, deterministic geometric code blocks.
  • Hard Constraints Unlock Accuracy: Enforcing invariants exactly does not restrict the model's capacity; instead, it provides a powerful inductive bias. By preventing the model from "shortcutting" through physically forbidden regions of the state space, accuracy is significantly improved.
  • Gradient Harmony: In multi-invariant systems, soft penalties often introduce conflicting gradient signals during backpropagation. By embedding constraints directly into the architecture, the optimization landscape becomes much cleaner.

Key Results

The compiled architectures consistently outperformed both unconstrained Neural ODEs and soft-penalty regularized baselines:

  • Constraint Satisfaction: In the highly nonlinear NOx reaction network, the compiled model reduced element conservation violations by over 104x compared to the best penalty baseline, keeping deviations at near-numerical precision ($\sim10^{-6}$).
  • Long-Horizon Extrapolation: When testing 10x past the training horizon on a thermomechanical system, the compiled model maintained a trajectory Mean Squared Error (MSE) below $10^{-6}$, performing 200x better than the standard Neural ODE.
  • Composability: For the two-body gravitational problem—which features a mix of analytically known momentum constraints and learned energy constraints—the compiler successfully unified them, yielding a 731x improvement in momentum tracking and a 6.9x reduction in total trajectory MSE.

Limitations and Future Work

While highly effective, the authors note several avenues for future development:

  • Constraint Scope: The current compiler focuses primarily on equality constraints and smooth manifolds. Future work will expand the framework to handle inequality constraints, contact dynamics, and discrete regime-switching events.
  • Scale and Dimensionality: Extending the compiler from Ordinary Differential Equations (ODEs) to Partial Differential Equations (PDEs) and stochastic systems is a key next step, alongside improving scalability for high-dimensional scientific problems.

Summary updated

Abstract

Neural ODEs are increasingly used as continuous-time models for scientific and sensor data, but unconstrained neural ODEs can drift and violate domain invariants (e.g., conservation laws), yielding physically implausible solutions. In turn, this can compound error in long-horizon prediction and surrogate simulation. Existing solutions typically aim to enforce invariance by soft penalties or other forms of regularization, which can reduce overall error but do not guarantee that trajectories will not leave the constraint manifold. We introduce the invariant compiler, a framework that enforces invariants by construction: it treats invariants as first-class types and uses an LLM-driven compilation workflow to translate a generic neural ODE specification into a structure-preserving architecture whose trajectories remain on the admissible manifold in continuous time (and up to numerical integration error in practice). This compiler view cleanly separates what must be preserved (scientific structure) from what is learned from data (dynamics within that structure). It provides a systematic design pattern for invariant-respecting neural surrogates across scientific domains.
BibTeX
@misc{yu2026invariant,
  author = {Fangzhou Yu and Yiqi Su and Ray Lee and Shenfeng Cheng and Naren Ramakrishnan},
  title = {{An Invariant Compiler for Neural ODEs in AI-Accelerated Scientific Simulation}},
  year = {2026},
  eprint = {2603.23861},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.23861},
}
Figure