Protein Folding with Neural Ordinary Differential Equations

Arielle Sanford, Shuo Sun, Christian B. Mendl

Introduction

AlphaFold 2 and its successors have fundamentally changed structural biology by providing atomic-accuracy protein structure predictions. However, this breakthrough relies on the "Evoformer," a massive neural engine typically composed of 48 discrete, stacked blocks. While effective, this architecture is computationally expensive, requiring industrial-scale TPU clusters for training and significant memory for inference. Furthermore, the rigid discretization of 48 layers applies the same computational depth to every protein, regardless of its structural complexity.

Researchers have recently proposed a more efficient alternative: the Neural ODE Evoformer. By reimagining the discrete steps of the Evoformer as a continuous-depth dynamical system, they have demonstrated that plausible protein structures can be predicted with a fraction of the resources, achieving training on a single consumer-grade GPU in under 18 hours.

The Evoformer ODE Architecture

The core innovation of this approach is treating protein folding as a smooth, continuous evolution rather than a series of discrete transformations. Instead of 48 separate layers with independent weights, the model defines a single "vector field"—a function that describes the infinitesimal change of the protein representation at any given point in the folding process.

Architecture of the Neural ODE Evoformer, illustrating the continuous flow of MSA and Pair representations through a learned vector field.

As shown in the architecture diagram, the model integrates this vector field over a continuous "depth" variable using a Neural ODE framework. Key technical modifications were made to ensure efficiency:

  • Adjoint Method: This technique allows for backpropagation with constant memory cost relative to depth, as intermediate activations do not need to be stored.
  • Simplified Operations: Memory-intensive components like "triangle attention" were replaced with symmetric approximations and simplified gating mechanisms.
  • Time-Dependent Gating: A small multi-layer perceptron (MLP) learns mixing coefficients ($\sigma_m, \sigma_z$) that modulate representation changes as the data progresses through the folding "time."

Performance and Scaling

One of the most significant advantages of the Neural ODE approach is its scaling behavior. While standard Evoformers typically exhibit quadratic scaling with protein length in terms of runtime, the Neural ODE implementation demonstrates a linear scaling trend. This shift makes the model a promising candidate for predicting the structures of very long protein chains that are currently computationally prohibitive for standard architectures.

Execution time comparison showing linear scaling for the Neural ODE approach versus quadratic scaling for the standard OpenFold implementation.

In empirical tests, the Neural ODE Evoformer averaged an inference speed of 4.85 seconds per protein, approximately 7 times faster per residue than the standard OpenFold implementation. Remarkably, the model was trained in just 17.5 hours on a single NVIDIA Quadro P4000 GPU, a stark contrast to the massive compute requirements of the original AlphaFold.

Structural Plausibility and Limitations

Qualitatively, the model produces structurally plausible results, showing particular strength in identifying and positioning $\alpha$-helices. Comparative analysis suggests that this continuous approximation is superior to simply truncating a discrete model (e.g., using only 24 blocks), as the ODE-based model produces more organized and confident structures.

However, as a proof-of-concept, an accuracy gap remains. The model currently struggles with fine-grained details in loop regions and complex global packing when compared to the full AlphaFold 2. These limitations are likely due to the simplified operations and the relatively small training set of 500 proteins used in this study.

Conclusion and Future Work

The Neural ODE Evoformer represents a shift away from "brute force" deep learning toward more efficient and adaptive models in structural biology. Future developments may include the use of adaptive solvers (like Dormand-Prince) to allow the model to dynamically allocate compute time based on protein difficulty, as well as scaling the training to larger, more diverse datasets. This work paves the way for high-quality structural predictions that are accessible to researchers without access to massive computing clusters.

Summary updated

BibTeX
@article{sanford2026protein,
  author = {Arielle Sanford and Shuo Sun and Christian B. Mendl},
  title = {{Protein Folding with Neural Ordinary Differential Equations}},
  journal = {Mach. Learn. Sci. Technol.},
  year = {2026},
  doi = {10.1088/2632-2153/ae5c55},
}
Figure