Geometric Hyena Networks for Large-scale Equivariant Learning

Artem Moskalev, Mangal Prakash, Junjie Xu, Tianyu Cui, Rui Liao, Tommaso Mansi

Abstract

Geometric Hyena (G-Hyena) is an equivariant long-convolutional model designed to process massive geometric graphs, such as large proteins and RNA molecules. By replacing quadratic-complexity self-attention with $O(N \log N)$ FFT-based "vector long convolutions," G-Hyena achieves up to a 20x speedup and handles context lengths 72x longer than state-of-the-art equivariant transformers. This architecture enables the processing of up to 2.7 million tokens on a single GPU, bridging the gap between the efficiency of local message passing and the global context of attention-based models.

1. Introduction

In the domains of structural biology and chemistry, molecules are represented as geometric graphs where nodes (atoms) possess specific coordinates in 3D space. To maintain physical consistency, models operating on these graphs must be equivariant: any rotation or translation of the input coordinates must result in a corresponding transformation of the model’s outputs, such as force vectors.

Current geometric deep learning architectures face a fundamental trade-off between efficiency and context. Local Message Passing Neural Networks (MPNNs) are computationally efficient but limited to local neighborhoods, often failing to capture critical long-range interactions. Conversely, Equivariant Transformers utilize self-attention to capture global context but suffer from $O(N^2)$ scaling. This quadratic complexity renders the modeling of large biological complexes, such as long RNA strands or multi-domain proteins, computationally prohibitive. G-Hyena addresses this by adapting sub-quadratic sequence models to the requirements of 3D geometric learning.

2. Methods

The core of the G-Hyena architecture is the adaptation of "long convolutions" to satisfy SE(3) equivariance (invariance to translation and equivariance to rotation).

2.1 Vector Long Convolutions

Standard long convolutions typically rely on dot products, which do not preserve rotation-equivariance for vector features. G-Hyena introduces a novel convolution operator based on the vector cross product. By decomposing this cross-product convolution into six parallel scalar convolutions, the model leverages the Fast Fourier Transform (FFT) to compute global interactions in $O(N \log N)$ time. This allows the model to maintain a global receptive field without the memory overhead of an attention matrix.

2.2 Invariant-Equivariant Interaction

G-Hyena maintains dual processing streams: a scalar stream for invariant features (e.g., atom types) and a vector stream for equivariant features (e.g., spatial coordinates). A "geometric long convolution" facilitates interaction between these streams, allowing the model to represent complex physical relationships, such as how scalar chemical properties influence directional forces.

2.3 Canonical Ordering and Gating

Unlike general point clouds, biological molecules like DNA, RNA, and proteins possess a natural "canonical" order defined by their polymer backbones. G-Hyena leverages this sequence structure to apply sequence-based operators effectively. Furthermore, the model incorporates selective gating, similar to the original Hyena architecture, allowing for dynamic filtering of global context based on the specific needs of each token.

3. Evaluation and Results

G-Hyena was evaluated across several large-scale benchmarks focusing on RNA and protein dynamics.

3.1 RNA Property Prediction

On the Open Vaccine COVID-19 and Ribonanza-2k datasets, which involve predicting the stability of RNA molecules up to 11,300 atoms, G-Hyena outperformed state-of-the-art models like Equiformer and VNT. The model achieved a reduction in Root Mean Square Error (RMSE) of up to 15%.

3.2 Molecular Dynamics

In the ProteinMD task, which predicts equilibrium trajectories of protein backbone atoms, G-Hyena reduced the final error by 9% compared to the strongest baseline. To further validate its capabilities, the authors introduced Geometric Associative Recall, a synthetic benchmark testing the model's ability to remember and rotate vector pairs across extended sequences.

3.3 Scaling and Efficiency

The primary advantage of G-Hyena is its scaling behavior. For sequences of 30,000 tokens, G-Hyena is 20x faster and utilizes 50x less memory than equivariant self-attention models. While traditional Transformers reached memory limits at approximately 37,000 tokens, G-Hyena successfully scaled to 2.7 million tokens on a single A10G GPU.

4. Discussion and Limitations

While G-Hyena represents a significant leap in scaling equivariant models, it relies on the existence of a canonical order. In scenarios involving unordered point clouds (e.g., LIDAR data), the model’s performance may be sensitive to the input sequence. Future work may explore learning canonical orderings or extending the architecture to higher-order steerable tensors (Type-2 and above) to capture more complex geometric symmetries.

5. Conclusion

Geometric Hyena Networks provide a scalable alternative to equivariant transformers by utilizing FFT-based vector convolutions. By treating biological molecules as sequences and maintaining SE(3) equivariance through cross-product interactions, G-Hyena enables the high-fidelity modeling of massive geometric structures that were previously beyond the reach of global-context models.

Summary updated

BibTeX
@inproceedings{moskalev2025geometric,
  author = {Artem Moskalev and Mangal Prakash and Junjie Xu and Tianyu Cui and Rui Liao and Tommaso Mansi},
  title = {{Geometric Hyena Networks for Large-scale Equivariant Learning}},
  booktitle = {International Conference on Machine Learning},
  year = {2025},
  url = {https://openreview.net/forum?id=jJRkkPr474},
}
Figure