D-CODE: Discovering Closed-form ODEs from Observed Trajectories

Zhaozhi Qian, Krzysztof Kacprzyk, Mihaela van der SchaarICLR 2022 · OpenReview

For centuries, scientists have sought to distill the complex dance of dynamical systems into elegant, closed-form ordinary differential equations (ODEs). While modern machine learning can "black-box" these systems with neural networks, we often lose the interpretability and analytical power that a concise formula provides. A paper from ICLR 2022, D-CODE, introduces a framework to automatically discover these governing equations directly from noisy, real-world observations.

TL;DR

D-CODE is a novel framework for discovering closed-form ODEs from observed trajectories without ever needing to estimate unobserved time derivatives. By leveraging a variational formulation, it transforms the discovery problem into a robust optimization task that significantly outperforms traditional symbolic regression in high-noise and low-sampling scenarios.

What Problem Does This Paper Address?

The core challenge in automated ODE discovery is the equation-data mismatch. Standard symbolic regression is a supervised learning tool: it needs "features" ($x$) and "labels" ($y$) to find a function $y = f(x)$. In the world of ODEs, the "label" is the time derivative $\dot{x}(t)$, but we almost never observe it. We only see the state $x(t)$ at discrete, often noisy, time points.

The traditional "two-step" approach is to estimate $\dot{x}(t)$ using numerical differentiation and then run symbolic regression. However, differentiation is a "noise-amplifier"—even tiny amounts of measurement error in the state can lead to wild, unusable estimates of the derivative.

High noise in derivative estimation leads to bias in traditional methods.

This makes existing methods fail when data is sparse or noisy, which is unfortunately the norm in fields like healthcare and biology.

Methods

The researchers' "aha moment" comes from the variational formulation of ODEs. Instead of trying to satisfy the equation $\dot{x}(t) = f(x(t))$ at every point, they look at the system through the lens of "testing functions" ($g$).

Overview of the D-CODE framework compared to traditional two-step symbolic regression.

By using integration by parts, the authors show that a trajectory $x$ satisfies an ODE $f$ if and only if a specific integral equals zero. Crucially, this integral involves the derivative of the testing function ($\dot{g}$), not the derivative of the noisy data ($\dot{x}$).

The D-CODE Algorithm: 1. Preprocessing: Smooth the noisy, discrete observations using a method like Gaussian Processes to get an estimated trajectory $\hat{x}$. 2. Optimization: Search for a closed-form function $f$ that minimizes a novel objective function based on the variational formulation. This objective acts as a proxy for the true estimation error. 3. Symbolic Search: The framework is "optimizer-agnostic," meaning it can work with various symbolic regression tools (the authors use Genetic Programming in their experiments).

Datasets

The authors evaluated D-CODE against several benchmarks across a variety of systems:

  • Synthetic Growth Models: The Gompertz and Generalized Logistic models (common in biology).
  • Biochemical Oscillators: The Glycolytic oscillator, which involves complex, non-linear interactions.
  • Chaotic Systems: The famous Lorenz system, testing the limits of the method on non-periodic, unpredictable data.
  • Real-world Clinical Data: A dataset of 310 trajectories of tumor volumes from cancer patients undergoing chemotherapy.

Key Insights and Ideas

  • Moving the Derivative: The most important insight is shifting the burden of differentiation from the noisy data to a smooth, mathematically defined testing function. This acts as a natural "noise filter."
  • Theoretical Justification: The authors provide a formal proof showing that as you increase the number of testing functions, the D-CODE objective converges to the squared distance between the candidate function and the true (but unknown) ODE.
  • Robustness to Artifacts: Unlike "two-step" methods that require the estimated derivative to converge to the truth, D-CODE only requires the trajectory to converge, which is a much easier condition to meet in practice.

Key Results

  • Noise Tolerance: In the Gompertz and Logistic models, D-CODE maintained a high "Success Probability" (recovering the exact functional form) even as noise levels increased, while benchmark methods' performance plummeted.
  • Handling Sparsity: D-CODE successfully recovered equations with sampling intervals up to 10x larger than what traditional methods could handle.
  • Chaotic Recovery: D-CODE was the only method tested that could reliably recover the governing equations of the chaotic Lorenz system under moderate noise.
  • Clinical Utility: On the tumor growth dataset, D-CODE discovered a concise equation that captured the "relapse" phase of tumor growth more accurately and with much lower error (RMSE 0.220 vs 0.637) than standard symbolic regression.

Limitations and Future Work

While powerful, the authors acknowledge several open challenges:

  • Unobserved Variables: If the system is influenced by external factors not recorded in the data, D-CODE can only provide an approximation.
  • Equation Complexity: Extremely long or high-dimensional equations remain difficult to search for due to the combinatorial explosion of the search space.
  • Extreme Settings: There is still a limit to how much noise or sparsity the system can handle before the signal is lost entirely.

The authors suggest that future work could extend this variational approach to discover Partial Differential Equations (PDEs) and Delay Differential Equations (DDEs), potentially opening new doors for automated discovery in physics and engineering.

Summary updated

Abstract

For centuries, scientists have manually designed closed-form ordinary differential equations (ODEs) to model dynamical systems. An automated tool to distill closed-form ODEs from observed trajectories would accelerate the modeling process. Traditionally, symbolic regression is used to uncover a closed-form prediction function $a=f(b)$ with label-feature pairs $(a_i, b_i)$ as training examples. However, an ODE models the time derivative $\dot{x}(t)$ of a dynamical system, e.g. $\dot{x}(t) = f(x(t),t)$, and the "label" $\dot{x}(t)$ is usually *not* observed. The existing ways to bridge this gap only perform well for a narrow range of settings with low measurement noise, frequent sampling, and non-chaotic dynamics. In this work, we propose the Discovery of Closed-form ODE framework (D-CODE), which advances symbolic regression beyond the paradigm of supervised learning. D-CODE leverages a novel objective function based on the variational formulation of ODEs to bypass the unobserved time derivative. For formal justification, we prove that this objective is a valid proxy for the estimation error of the true (but unknown) ODE. In the experiments, D-CODE successfully discovered the governing equations of a diverse range of dynamical systems under challenging measurement settings with high noise and infrequent sampling.
BibTeX
@inproceedings{qian2022d,
  author = {Zhaozhi Qian and Krzysztof Kacprzyk and Mihaela van der Schaar},
  title = {{D-CODE: Discovering Closed-form ODEs from Observed Trajectories}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=VavfaCVE8oW},
}
Figure