For centuries, scientists have sought to distill the complex dance of dynamical systems into elegant, closed-form ordinary differential equations (ODEs). While modern machine learning can "black-box" these systems with neural networks, we often lose the interpretability and analytical power that a concise formula provides. A paper from ICLR 2022, D-CODE, introduces a framework to automatically discover these governing equations directly from noisy, real-world observations.
TL;DR
D-CODE is a novel framework for discovering closed-form ODEs from observed trajectories without ever needing to estimate unobserved time derivatives. By leveraging a variational formulation, it transforms the discovery problem into a robust optimization task that significantly outperforms traditional symbolic regression in high-noise and low-sampling scenarios.
What Problem Does This Paper Address?
The core challenge in automated ODE discovery is the equation-data mismatch. Standard symbolic regression is a supervised learning tool: it needs "features" ($x$) and "labels" ($y$) to find a function $y = f(x)$. In the world of ODEs, the "label" is the time derivative $\dot{x}(t)$, but we almost never observe it. We only see the state $x(t)$ at discrete, often noisy, time points.
The traditional "two-step" approach is to estimate $\dot{x}(t)$ using numerical differentiation and then run symbolic regression. However, differentiation is a "noise-amplifier"—even tiny amounts of measurement error in the state can lead to wild, unusable estimates of the derivative.
This makes existing methods fail when data is sparse or noisy, which is unfortunately the norm in fields like healthcare and biology.
Methods
The researchers' "aha moment" comes from the variational formulation of ODEs. Instead of trying to satisfy the equation $\dot{x}(t) = f(x(t))$ at every point, they look at the system through the lens of "testing functions" ($g$).
By using integration by parts, the authors show that a trajectory $x$ satisfies an ODE $f$ if and only if a specific integral equals zero. Crucially, this integral involves the derivative of the testing function ($\dot{g}$), not the derivative of the noisy data ($\dot{x}$).
The D-CODE Algorithm: 1. Preprocessing: Smooth the noisy, discrete observations using a method like Gaussian Processes to get an estimated trajectory $\hat{x}$. 2. Optimization: Search for a closed-form function $f$ that minimizes a novel objective function based on the variational formulation. This objective acts as a proxy for the true estimation error. 3. Symbolic Search: The framework is "optimizer-agnostic," meaning it can work with various symbolic regression tools (the authors use Genetic Programming in their experiments).
Datasets
The authors evaluated D-CODE against several benchmarks across a variety of systems:
- Synthetic Growth Models: The Gompertz and Generalized Logistic models (common in biology).
- Biochemical Oscillators: The Glycolytic oscillator, which involves complex, non-linear interactions.
- Chaotic Systems: The famous Lorenz system, testing the limits of the method on non-periodic, unpredictable data.
- Real-world Clinical Data: A dataset of 310 trajectories of tumor volumes from cancer patients undergoing chemotherapy.
Key Insights and Ideas
- Moving the Derivative: The most important insight is shifting the burden of differentiation from the noisy data to a smooth, mathematically defined testing function. This acts as a natural "noise filter."
- Theoretical Justification: The authors provide a formal proof showing that as you increase the number of testing functions, the D-CODE objective converges to the squared distance between the candidate function and the true (but unknown) ODE.
- Robustness to Artifacts: Unlike "two-step" methods that require the estimated derivative to converge to the truth, D-CODE only requires the trajectory to converge, which is a much easier condition to meet in practice.
Key Results
- Noise Tolerance: In the Gompertz and Logistic models, D-CODE maintained a high "Success Probability" (recovering the exact functional form) even as noise levels increased, while benchmark methods' performance plummeted.
- Handling Sparsity: D-CODE successfully recovered equations with sampling intervals up to 10x larger than what traditional methods could handle.
- Chaotic Recovery: D-CODE was the only method tested that could reliably recover the governing equations of the chaotic Lorenz system under moderate noise.
- Clinical Utility: On the tumor growth dataset, D-CODE discovered a concise equation that captured the "relapse" phase of tumor growth more accurately and with much lower error (RMSE 0.220 vs 0.637) than standard symbolic regression.
Limitations and Future Work
While powerful, the authors acknowledge several open challenges:
- Unobserved Variables: If the system is influenced by external factors not recorded in the data, D-CODE can only provide an approximation.
- Equation Complexity: Extremely long or high-dimensional equations remain difficult to search for due to the combinatorial explosion of the search space.
- Extreme Settings: There is still a limit to how much noise or sparsity the system can handle before the signal is lost entirely.
The authors suggest that future work could extend this variational approach to discover Partial Differential Equations (PDEs) and Delay Differential Equations (DDEs), potentially opening new doors for automated discovery in physics and engineering.
Summary updated