Frame Averaging for Invariant and Equivariant Network Design

Omri Puny, Matan Atzmon, Heli Ben-Hamu, Edward J. Smith, Ishan Misra, Aditya Grover, Yaron Lipman

Abstract

Many machine learning tasks involve learning functions that are known to be invariant or equivariant to certain symmetries of the input data. However, it is often challenging to design neural network architectures that respect these symmetries while being expressive and computationally efficient. We introduce Frame Averaging (FA), a general purpose and systematic framework for adapting known (backbone) architectures to become invariant or equivariant to new symmetry types. Our framework builds on the well known group averaging operator but replaces it with an averaging operator over a small subset of the group elements, called a frame. We show that FA-based models have maximal expressive power and demonstrate their effectiveness on point cloud normal estimation, graph separation, and n-body dynamics prediction, achieving state-of-the-art results in all of these benchmarks.

Introduction

In many machine learning domains—like 3D molecular modeling, robotics, or social network analysis—the data has inherent symmetries. For example, a molecule’s properties shouldn't change if you rotate it (invariance), and a robot’s predicted path should rotate exactly as the robot does (equivariance). Designing neural networks that respect these symmetries is notoriously difficult, often forcing a choice between mathematically complex hand-crafted layers, inefficient data augmentation, or computationally intractable group averaging. Frame Averaging (FA) provides a systematic "plug-and-play" solution that preserves the expressivity of standard backbone architectures while guaranteeing exact symmetry.

The Frame Averaging Framework

The core of FA is the replacement of the intractable average over an entire symmetry group $G$ with an average over a data-dependent, equivariant subset called a frame.

Conceptual diagram of Frame Averaging mapping a group $G$ to a vector space $V$ via an equivariant frame $\mathcal{F}$.

A frame $\mathcal{F}(X) \subseteq G$ is chosen based on the input $X$ such that if the input is transformed by $g \in G$, the frame transforms accordingly: $\mathcal{F}(g \cdot X) = g \cdot \mathcal{F}(X)$. By averaging the output of a standard backbone model (like PointNet or an MLP) over the elements of this frame, the resulting function is guaranteed to be invariant or equivariant. For point clouds, a frame can be efficiently constructed using Principal Component Analysis (PCA), resulting in a small set of just 8 rotations/reflections in 3D, making the process computationally efficient.

Expressivity and Universality

A significant advantage of FA is that it preserves the expressive power of the underlying backbone. If the original model is a universal approximator, the frame-averaged version becomes a universal symmetric approximator. This allows for the creation of "Universal GNNs" by using the eigenvectors of the Graph Laplacian to define frames. This approach enables the distinction of graphs that standard message-passing neural networks (like GIN) cannot separate, effectively going beyond the 2-WL graph isomorphism test.

Experimental Results

The effectiveness of FA was evaluated across point cloud, graph, and physical dynamics benchmarks.

Point Cloud Normal Estimation

On the ABC dataset of CAD models, FA-enhanced versions of PointNet and DGCNN were tested for their ability to predict surface normals under various rotation settings.

Performance comparison of standard and FA-based models on point cloud normal estimation across different rotation settings (I/I, I/SO(3), and SO(3)/SO(3)).

As shown in the results, FA models significantly outperform standard versions, particularly in the "I/SO(3)" setting (trained on fixed data, tested on rotated data). While standard models fail to generalize to unseen rotations, FA models maintain high accuracy due to their inherent symmetry guarantees. FA-DGCNN, in particular, achieves state-of-the-art performance across multiple sample sizes.

Graph Separation and Physics

FA-based models achieved 100% accuracy on the EXP graph separation dataset, where standard GNNs like GCN, GAT, and GIN only achieved 50% (equivalent to random guessing). In n-body dynamics prediction, FA-GNN achieved a Mean Squared Error (MSE) of 0.0057, representing a 20% improvement over the previous state-of-the-art (EGNN) while maintaining similar inference speeds.

Conclusion

Frame Averaging offers a systematic and efficient way to build symmetric neural networks. By leveraging small, data-dependent frames, it avoids the computational pitfalls of full group averaging while maintaining the full expressive power of modern deep learning architectures. Its success across diverse benchmarks suggests it is a powerful general-purpose tool for geometric deep learning and beyond.

Summary updated

Abstract

Many machine learning tasks involve learning functions that are known to be invariant or equivariant to certain symmetries of the input data. However, it is often challenging to design neural network architectures that respect these symmetries while being expressive and computationally efficient. For example, Euclidean motion invariant/equivariant graph or point cloud neural networks. We introduce Frame Averaging (FA), a general purpose and systematic framework for adapting known (backbone) architectures to become invariant or equivariant to new symmetry types. Our framework builds on the well known group averaging operator that guarantees invariance or equivariance but is intractable. In contrast, we observe that for many important classes of symmetries, this operator can be replaced with an averaging operator over a small subset of the group elements, called a frame. We show that averaging over a frame guarantees exact invariance or equivariance while often being much simpler to compute than averaging over the entire group. Furthermore, we prove that FA-based models have maximal expressive power in a broad setting and in general preserve the expressive power of their backbone architectures. Using frame averaging, we propose a new class of universal Graph Neural Networks (GNNs), universal Euclidean motion invariant point cloud networks, and Euclidean motion invariant Message Passing (MP) GNNs. We demonstrate the practical effectiveness of FA on several applications including point cloud normal estimation, beyond $2$-WL graph separation, and $n$-body dynamics prediction, achieving state-of-the-art results in all of these benchmarks.
BibTeX
@inproceedings{puny2022frame,
  author = {Omri Puny and Matan Atzmon and Heli Ben-Hamu and Edward J. Smith and Ishan Misra and Aditya Grover and Yaron Lipman},
  title = {{Frame Averaging for Invariant and Equivariant Network Design}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=z1nUI70ADKJ},
}
Figure