Training Dynamics of Learning 3D-Rotational Equivariance

Max W. Shen, Ewa Nowara, Michael Maser, Kyunghyun Cho

TL;DR

When trained with simple data augmentation, standard neural networks learn 3D-rotational equivariance remarkably fast—often within the first few minutes of training. This happens because learning symmetry is a fundamentally "easier" optimization problem characterized by a significantly smoother and better-conditioned loss landscape than the main prediction task.

What Problem Does This Paper Address?

In molecular machine learning, capturing physical symmetries like 3D rotation is crucial. Historically, researchers have faced a stark choice: use mathematically rigorous, symmetry-preserving (equivariant) architectures, or use standard, symmetry-agnostic models trained with data augmentation (randomly rotating inputs).

While equivariant architectures are elegant, they carry a massive "efficiency gap"—they can be 10x to 100x slower to compute and harder to optimize. Conversely, data augmentation is fast, but it has been unclear exactly how well or how quickly symmetry-agnostic models actually learn to respect these vital geometric symmetries. This paper introduces a principled framework to quantify this exact phenomenon.

Methods

The authors propose a formal framework to isolate and measure a model's equivariance error.

  1. Twisting and Twirling: To evaluate a model $f$, they introduce a "twisted prediction" by rotating an input, passing it through the model, and un-rotating the output back to a canonical frame. Averaging these twisted predictions over all possible rotations yields the "twirled prediction," which is perfectly equivariant by design.
  2. Exact Loss Decomposition: For convex loss functions like Mean Squared Error (MSE), the authors mathematically prove that the total validation loss decomposes cleanly into two independent components: $$\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{mean}} \text{ (prediction error)} + \mathcal{L}_{\text{equiv}} \text{ (equivariance error)}$$
  3. Loss Landscape Analysis: They compute the Hessian matrices of both loss components at early training checkpoints to evaluate the smoothness and conditioning (via condition numbers) of their respective optimization landscapes.
Figure 1: Overview of the paper. (a) Schematic of twisting and twirling, which underpin a principled measure of equivariance error. (b) Loss decomposition by Taylor expansion around the twirled prediction. (c) Loss landscapes for each loss component at early model checkpoints (step=500). (d) Architectures of three non-equivariant models studied here. (e) For MSE loss, the loss decomposition holds exactly, enabling computing the percent validation loss from equivariance error, which is plotted by training step in three settings.

Datasets

The framework was empirically validated across three distinct high-dimensional molecular tasks:

  • SPICE: A dataset of small molecule 3D conformers with quantum-mechanical properties, used to evaluate force field prediction via the EScAIP graph transformer.
  • Protein Data Bank (PDB): Used for a generative probabilistic flow matching task via Proteína (an AlphaFold3-like transformer).
  • GEOM-drugs: A dataset of 3D drug-like molecular structures, used for a voxelized denoising autoencoder task via VoxMol (a 3D CNN).

Key Insights and Ideas

  • Equivariance is an "Easy" Sub-task: The most striking finding is that the loss landscape for the equivariance error ($\mathcal{L}_{\text{equiv}}$) is dramatically smoother and better-conditioned than the main prediction task ($\mathcal{L}_{\text{mean}}$). Its condition number is 10x to 1000x smaller, explaining why the network resolves its internal symmetry conflicts so rapidly.
  • The Robustness of Symmetry Learning: The rapid drop in equivariance error is remarkably robust. It holds true regardless of dramatic changes to model size (from 1M to 400M parameters), dataset scale (from 1M down to 500 samples), batch sizes, or choice of optimizer (Adam vs. SGD).
  • The Efficiency Tradeoff: Because the loss penalty for non-equivariance becomes negligible so early in training, standard models with data augmentation often deliver a better test-loss-per-GPU-hour return on investment than strictly equivariant models.

Key Results

  • The 2% Threshold: Across all models and datasets, the equivariance error plummets to $\le$ 2% of the total held-out validation loss within just 1,000 to 10,000 training steps.
  • Quantified Landscapes: For the EScAIP model, the condition number for the prediction loss was roughly $10^9$, whereas the equivariance loss was $10^6$ (1,000x smaller). For Proteína, it was 100x smaller ($2\times10^{10}$ vs $10^8$).
  • Parameter Alignment: The authors proved and empirically verified a tight quadratic relationship (Spearman correlation = 0.99) between the model's actual equivariance error and its parameter distance from a perfectly equivariant weight subspace.
Figure 2: Training dynamics of learning equivariance in EScAIP (Force field prediction). (a-c) Validation losses and percent validation loss from equivariance error during training, early in training (a), with log-log axes (b), and decomposed into separate terms (c). (d-f) Impact of varying training set size (d), model size (e), and optimizer or learning rate (f).

Limitations and Future Work

  • Symmetry Scope: This study focuses strictly on 3D rotations. Other physically relevant symmetries, such as translations or discrete permutations (e.g., graph isomorphism), may exhibit entirely different training dynamics.
  • Latent Space Disconnect: While the outputs of these models become highly equivariant, the authors noted that the internal latent representations of models like VoxMol do not automatically become equivariant or invariant, suggesting that architectural constraints might still be necessary if downstream tasks rely heavily on structured latents.
  • Closing the Remaining Gap: Even though the error drops below 2%, it rarely hits absolute zero. Future work will explore whether simple test-time post-processing (like test-time twirling) can cheaply eliminate this residual error entirely.

Summary updated

Abstract

While data augmentation is widely used to train symmetry-agnostic models, it remains unclear how quickly and effectively they learn to respect symmetries. We investigate this by deriving a principled measure of equivariance error that, for convex losses, calculates the percent of total loss attributable to imperfections in learned symmetry. We focus our empirical investigation to 3D-rotation equivariance on high-dimensional molecular tasks (flow matching, force field prediction, denoising voxels) and find that models reduce equivariance error quickly to $\leq$2\% held-out loss within 1k-10k training steps, a result robust to model and dataset size. This happens because learning 3D-rotational equivariance is an easier learning task, with a smoother and better-conditioned loss landscape, than the main prediction task. For 3D rotations, the loss penalty for non-equivariant models is small throughout training, so they may achieve lower test loss than equivariant models per GPU-hour unless the equivariant ``efficiency gap'' is narrowed. We also experimentally and theoretically investigate the relationships between relative equivariance error, learning gradients, and model parameters.
BibTeX
@inproceedings{shen2025training,
  author = {Max W. Shen and Ewa Nowara and Michael Maser and Kyunghyun Cho},
  title = {{Training Dynamics of Learning 3D-Rotational Equivariance}},
  booktitle = {AI4Science @ NeurIPS},
  year = {2025},
  url = {https://openreview.net/forum?id=Yddq1QmIPK},
}
Figure