Scalars are universal: Equivariant machine learning, structured like classical physics

Soledad Villar, David W Hogg, Kate Storey-Fisher, Weichi Yao, Ben Blum-Smith

In the quest to build machine learning models that inherently "understand" the physical world, researchers have long grappled with the concept of equivariance—the property where a model’s output transforms in a predictable, consistent way when its input is transformed (e.g., rotated, shifted, or boosted). While Convolutional Neural Networks (CNNs) elegantly handle spatial translations, managing rotations, reflections, and relativistic boosts typically requires heavy-duty group theory.

The paper "Scalars are universal: Equivariant machine learning, structured like classical physics" provides a refreshing, physics-inspired alternative. Instead of relying on complex mathematical decompositions, the authors demonstrate that universal equivariant models can be built using nothing more than simple scalar products.

The Challenge with Traditional Equivariant Networks

Most physical laws are equivariant by nature. If you rotate a system of interacting particles, the forces acting upon them rotate by the exact same amount. Standard neural networks are oblivious to this geometric reality; they must "learn" symmetries through massive data augmentation, which is inefficient and prone to error.

To address this, the field developed Equivariant Neural Networks. However, the state-of-the-art approaches typically rely on irreducible representations (irreps) and Clebsch-Gordan coefficients. While mathematically rigorous, these methods suffer from three major drawbacks:

  1. Complexity: They require deep expertise in group representation theory.
  2. Rigidity: Implementations are often hard-coded for specific dimensionalities (like 2D or 3D) and do not generalize easily.
  3. Computational Overhead: Decomposing tensors into irreps is computationally expensive.

The authors ask a fundamental question: Is there a simpler way to achieve universal equivariance that mirrors how physicists actually write equations?

The Core Idea: Scalars are Universal

The core methodology is rooted in the First Fundamental Theorem of Invariant Theory. The authors prove that any physical law respecting standard symmetries—such as rotations, translations, or Lorentz boosts—can be represented as a function of invariant scalars.

Specifically, for groups like $O(d)$ (orthogonal/rotations) and $O(1, d)$ (Lorentz/relativity), an equivariant vector function $h(v_1, \dots, v_n)$ can always be expressed as:

$$h(v_1, \dots, v_n) = \sum_{t=1}^{n} f_t(\langle v_i, v_j \rangle) v_t$$

In practical terms, to generate an equivariant vector output, the model simply computes a weighted sum of the input vectors. The "weights" ($f_t$) are standard neural networks (like MLPs) whose inputs are the scalar products (dot products) of all pairs of input vectors.

This logic elegantly extends to other symmetries:

  • Euclidean and Poincaré groups: Handled by using the differences between position vectors.
  • Permutation symmetry ($S_n$): Achieved by ensuring the scalar functions $f_t$ treat the set of inputs as exchangeable, utilizing architectures like DeepSets.

Experimental Validation

The authors evaluated their scalar-based method on two physically grounded learning tasks, extending the Equivariant Multi-Layer Perceptron (EMLP) framework:

  1. $O(5)$-invariant task: Learning a complex scalar function of two 5D vectors.
  2. $O(3)$-equivariant task: Predicting a moment-of-inertia-like matrix from the positions and masses of five particles, requiring the model to simultaneously respect rotation, reflection, and permutation symmetries.

Key Results: * Superior Accuracy: The scalar-based model achieved significantly lower test Mean Squared Error (MSE) than standard MLPs, augmented MLPs, and even irrep-based EMLPs. * Sample Efficiency: Because the symmetry is a mathematical guarantee rather than a learned approximation, the model generalizes perfectly to unseen rotations and reaches high accuracy with far fewer training examples. * Dimensionality Agnostic: Unlike irrep-based methods that break or require complete rewrites when moving from 3D to 4D, the scalar-based method works seamlessly for any dimension $d$ because the dot product is universally defined.

Limitations and Future Directions

While the scalar-based approach is powerful, the authors note a few current limitations and areas for future research:

  • The Scalar Explosion: For systems with thousands of particles, computing all $n^2$ dot products becomes a computational bottleneck. However, because only $O(dn)$ scalars are actually independent, the authors suggest using "rigidity theory" to select a minimal necessary subset.
  • Global vs. Local Symmetries: The current framework focuses on global symmetries (e.g., rotating the entire system). It does not yet account for gauge symmetries (local coordinate changes), though incorporating concepts like parallel transport could bridge this gap.
  • Dimensional Analysis: The model respects geometric symmetries but remains ignorant of physical units (e.g., it might mathematically attempt to add meters to seconds). Integrating dimensional analysis into the scalar framework remains an exciting frontier.

By structuring machine learning models like classical physics equations, this research offers a highly scalable, intuitive, and efficient path forward for scientific AI.

Summary updated

Abstract

There has been enormous progress in the last few years in designing neural networks that respect the fundamental symmetries and coordinate freedoms of physical law. Some of these frameworks make use of irreducible representations, some make use of high-order tensor objects, and some apply symmetry-enforcing constraints. Different physical laws obey different combinations of fundamental symmetries, but a large fraction (possibly all) of classical physics is equivariant to translation, rotation, reflection (parity), boost (relativity), and permutations. Here we show that it is simple to parameterize universally approximating polynomial functions that are equivariant under these symmetries, or under the Euclidean, Lorentz, and Poincaré groups, at any dimensionality $d$. The key observation is that nonlinear O($d$)-equivariant (and related-group-equivariant) functions can be universally expressed in terms of a lightweight collection of scalars---scalar products and scalar contractions of the scalar, vector, and tensor inputs. We complement our theory with numerical examples that show that the scalar-based method is simple, efficient, and scalable.
BibTeX
@inproceedings{villar2021scalars,
  author = {Soledad Villar and David W Hogg and Kate Storey-Fisher and Weichi Yao and Ben Blum-Smith},
  title = {{Scalars are universal: Equivariant machine learning, structured like classical physics}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=NqYtJMX9g2t},
}
Figure