On the Universality of Rotation Equivariant Point Cloud Networks

Nadav Dym, Haggai Maron

TL;DR

This paper provides the first theoretical proof that certain rotation-equivariant neural networks for 3D point clouds are "universal approximators." The authors demonstrate that popular architectures like Tensor Field Networks (TFN) can approximate any continuous equivariant function and introduce a framework to design new, simpler architectures that maintain this same mathematical power.

What Problem Does This Paper Address?

When processing 3D point clouds (like LIDAR scans or molecular structures), we want our neural networks to respect the laws of physics. Specifically, if you rotate, translate, or reorder the points in a cloud, the network's output should change in a predictable, "equivariant" way. For example, if an input molecule rotates 90 degrees, its predicted force vectors should also rotate 90 degrees.

While researchers have developed many "equivariant" architectures, a critical theoretical question remained: Are these networks universal? In other words, is there any valid physical function that these networks cannot learn, no matter how large they are?

The theoretical gap between equivariant functions and networks

While some models (like standard Graph Neural Networks) are known to have "blind spots," this paper investigates whether rotation-equivariant point cloud networks suffer from similar limitations, aiming to close the theoretical gap between the space of all continuous equivariant functions and the functions expressible by equivariant networks.

Methods

The authors develop a general framework for proving universality based on two sufficient conditions:

  1. D-spanning: The network must be able to represent a specific set of equivariant polynomials. This is based on a novel characterization of the space of polynomials that are invariant to translations and permutations but equivariant to rotations.
  2. Linear Universality: The network must be able to express all possible linear equivariant mappings between its internal data representations.

Using this framework, they analyze two main approaches:

  • Tensor Field Networks (TFN): They prove that this popular architecture, which uses "irreducible representations" (irreps) and spherical harmonics, is indeed universal.
  • Tensor Product Representations: They propose a "Minimal Universal Architecture" that uses simpler tensor products instead of the complex machinery of irreps, showing that universality can be achieved with a more straightforward (though more redundant) data structure.

Crucially, they show that translation invariance can be handled simply by "centralizing" the point cloud (subtracting the mean), allowing the theory to focus purely on rotation and permutation.

Key Insights and Ideas

  • The Power of "Lifting": To be universal, a network must "lift" the data into higher-dimensional representations. In TFNs, this means using higher-order spherical harmonics (higher $L$ values).
Lifting input data to higher-dimensional representations
  • Non-linearities are Optional (Theoretically): One of the most surprising findings is that standard non-linear activation functions (like ReLU) are not strictly necessary for universality in these architectures. The "lifting" process and the way tensor products are combined provide enough expressive power to approximate any continuous function.
  • Polynomials as the Bridge: The proof relies on showing that if a network can represent certain equivariant polynomials, it can approximate any continuous equivariant function (a variation of the Stone-Weierstrass theorem).

Key Results and Empirical Validation

The authors evaluated their theoretical findings using the QM9 dataset, a standard benchmark containing roughly 134,000 small molecules. The task involved predicting molecular properties (specifically the homo target variable, which relates to electron energy levels) based on 3D atomic positions and features.

  • Theoretical Proof: The paper successfully proves that TFNs and their variants are universal approximators for 3D point clouds.
  • Empirical Validation: On the QM9 dataset, the authors found that increasing the "order" of the representation (from $L=0$ to $L=4$) significantly improved accuracy, with the Mean Absolute Error dropping from 111.2 to 51.2.
  • Simplification Works: Their experiments confirmed that removing non-linearities or simplifying the "self-interaction" layers had a negligible impact on performance, validating their theory that the architectural structure itself carries the bulk of the expressive power.

Limitations and Future Work

  • Computational Cost: While using high-order representations (high $L$) guarantees universality, it is computationally expensive and memory-intensive. Practical implementations often stick to low orders ($L=1$ or $2$).
  • The "Low-Dimensional" Gap: An open question remains: Can we achieve universality using only low-dimensional representations? The current proof requires high-order tensors to "span" the space of possible functions.
  • Generalization: While the paper focuses on 3D space ($SO(3)$), the authors suggest their framework could be generalized to other symmetry groups and higher dimensions.

Summary updated

Abstract

Learning functions on point clouds has applications in many fields, including computer vision, computer graphics, physics, and chemistry. Recently, there has been a growing interest in neural architectures that are invariant or equivariant to all three shape-preserving transformations of point clouds: translation, rotation, and permutation. In this paper, we present a first study of the approximation power of these architectures. We first derive two sufficient conditions for an equivariant architecture to have the universal approximation property, based on a novel characterization of the space of equivariant polynomials. We then use these conditions to show that two recently suggested models, Tensor field Networks and SE3-Transformers, are universal, and for devising two other novel universal architectures.
BibTeX
@inproceedings{dym2021universality,
  author = {Nadav Dym and Haggai Maron},
  title = {{On the Universality of Rotation Equivariant Point Cloud Networks}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
  url = {https://openreview.net/forum?id=CzmiO1pC8Ao},
}
Figure