Neural Ordinary Differential Equation Models of Circuits: Capabilities and Pitfalls

Jie Xiong, Alan Yang, Maxim Raginsky, Elyse RosenbaumIEEE Transactions on Microwave Theory and Techniques 2022 · DOI

TL;DR

This paper advances the use of Neural Ordinary Differential Equations (ODEs) to create fast, continuous-time behavioral models for electronic circuits. By introducing specialized architectures for oscillators and "stiff" systems and providing a rigorous framework for Input-to-State Stability (ISS), the authors demonstrate how to build models that are not only 100x faster than transistor-level simulations but also guaranteed to behave physically when connected to other components.

What Problem Does This Paper Address?

In the world of integrated circuit design, engineers face a constant trade-off between accuracy and speed. Transistor-level simulations (like SPICE) are highly accurate but painfully slow for large systems. Behavioral models—mathematical "black boxes" that mimic circuit behavior—offer a faster alternative.

However, traditional neural network-based behavioral models face two major hurdles:

  1. Time-Step Mismatch: Most prior models are discrete-time. When a circuit simulator uses an adaptive time step (changing the step size to capture fast transitions), discrete models lose accuracy.
  2. Stability Hazards: Circuits are rarely used in isolation. When you connect a model to a source or a load, you often inadvertently create feedback loops. If the model isn't mathematically "stable," these loops can trigger nonphysical oscillations or cause the simulation to crash.

Methods

The authors propose using Neural ODEs, where the time derivatives of the circuit's state are parameterized by a neural network. This makes the models inherently continuous-time and compatible with the adaptive solvers used in standard EDA tools.

Key technical contributions include:

  • Specialized Architectures: Beyond standard Continuous-Time Recurrent Neural Networks (CTRNNs), they introduce the CTGRU (Continuous-Time Gated Recurrent Unit) for "stiff" circuits with long-term dependencies and the ONODE (Oscillatory Neural ODE) specifically designed to model oscillators.
  • Stability by Design: They utilize a reparameterization technique to enforce Input-to-State Stability (ISS) during training. This ensures the model's internal states remain bounded for any bounded input.
  • System-Level Stability: They apply the Small-Gain Theorem to derive conditions under which a model remains stable when interconnected with other circuit blocks (like nonideal sources or RC loads).
  • Tool Integration: The models are implemented in Verilog-A, allowing them to be dropped directly into professional SPICE environments.

Datasets

The models were trained and evaluated using data generated from SPICE simulations of five distinct circuit types:

  • Inverter Chain: A standard benchmark for signal propagation.
  • Flip-Flop: A bistable circuit used to test the model's ability to handle memory states.
  • Large IP Block: A complex circuit with ~2000 transistors to demonstrate scaling and speedup.
  • I/Q Clock Generator: Used to test frequency-domain characteristics.
  • Voltage-Controlled Ring Oscillator (VCRO): A 5-stage oscillator used to validate the ONODE architecture.

Key Insights and Ideas

  • The Continuous Advantage: Neural ODEs are a "natural" fit for circuits. Unlike discrete RNNs, they don't "break" when the simulator decides to take a tiny time step to capture a sharp edge.
  • Architecture is Destiny: Standard RNNs often fail to model oscillators, treating the periodic signal as "noise" and settling into a constant output. The ONODE architecture solves this by constraining parameters to maintain a periodic trajectory.
  • Interconnection is the Real Test: A model that is stable on its own can become unstable when connected to a load. The authors show that carefully choosing whether a port is modeled as a voltage-in/current-out ($I(V)$) or current-in/voltage-out ($V(I)$) can eliminate feedback loops and bypass the need for conservative stability constraints.
  • Solver Choice Matters: During training, using a higher-order solver like RK4 (Runge-Kutta 4th order) significantly improves the final model's accuracy in SPICE compared to simple Euler integration.

Key Results

  • Massive Speedups: For the 2000-transistor IP block, the Neural ODE model was over 100x faster than the original transistor-level netlist.
  • Accuracy: Models trained with RK4 achieved very low NRMSE (Normalized Root Mean Square Error), often below 2-3% in validation.
  • Robustness: ISS-constrained models successfully avoided the nonphysical "blow-ups" that unconstrained models sometimes exhibited when faced with stimuli outside their training set.

Limitations and Future Work

  • Conservative Bounds: The stability conditions derived from the Small-Gain Theorem are "sufficient but not necessary." This means they can be overly restrictive, potentially limiting the model's accuracy to guarantee stability.
  • The Curse of Dimensionality: Training data requirements grow exponentially with the number of I/O ports. Modeling a 64-bit bus as a single Neural ODE is currently infeasible.
  • Future Direction: The authors suggest breaking large systems into smaller, interconnected sub-circuits. If these sub-circuits are stable by construction, the entire system-level model can be built hierarchically without losing the speed advantages of the Neural ODE approach.

Summary updated

Abstract

This work advances the application of neural ordinary differential equations (ODEs) to circuit modeling. Prior works primarily utilized the recurrent neural network (RNN), which is a specific type of neural ODE. In this work, the capability of neural ODEs to represent different types of circuits is studied. Stability conditions are presented, both for neural ODEs in a standalone configuration and for neural ODEs with feedback connections, and practical techniques to impose the stability constraints during training are demonstrated. Based on the theoretical and experimental results, this work provides guidance as to when and how an accurate and stable neural ODE circuit model can be generated.
BibTeX
@article{xiong2022neural,
  author = {Jie Xiong and Alan Yang and Maxim Raginsky and Elyse Rosenbaum},
  title = {{Neural Ordinary Differential Equation Models of Circuits: Capabilities and Pitfalls}},
  journal = {IEEE Transactions on Microwave Theory and Techniques},
  year = {2022},
  doi = {10.1109/tmtt.2022.3208896},
}
Figure