Point Edge Transformer model trained on highly distorted methane configurations

Sergey N. Pozdnyakov, Michele CeriottiarXiv 2023 · DOI · arXiv

TL;DR

Researchers have developed a general protocol called ECSE (Equivariant Coordinate System Ensemble) that can turn almost any point-cloud neural network into a physically consistent, rotationally equivariant model. By applying this method to a new architecture called the Point Edge Transformer (PET), they achieved state-of-the-art accuracy across diverse chemical datasets, proving that we don't need to sacrifice architectural flexibility to satisfy the laws of physics.

What Problem Does This Paper Address?

In the world of 3D deep learning, "point clouds" (collections of points in space) are everywhere—from LiDAR scans for self-driving cars to the arrangement of atoms in a protein. However, the requirements for these two domains are very different.

In computer vision, "close enough" is often fine. But in chemistry and materials science, models must strictly obey physical symmetries: if you rotate a molecule, its predicted energy shouldn't change (invariance), and its predicted forces should rotate exactly with it (equivariance). Most general-purpose models (like PointNet++) ignore these constraints, while "physics-first" models are often restricted to specific, complex mathematical forms that can limit their accuracy or make them harder to develop.

The authors tackle this "wall" between communities, seeking a way to use the most powerful, flexible architectures from general computer vision while guaranteeing the rigorous symmetry required for scientific simulations.

Methods

The paper introduces two major contributions:

  1. ECSE (Equivariant Coordinate System Ensemble): This is a "wrapper" method. Instead of building a model that is inherently equivariant, you take a standard, non-equivariant model and look at the input from many different local perspectives. Specifically, for every atom, the method defines a coordinate system based on pairs of its neighbors. It then averages the model’s predictions across all these perspectives. To prevent "glitches" (discontinuities) when atoms move in or out of range, they use a sophisticated smooth weighting system and an adaptive cutoff.
  2. PET (Point Edge Transformer): To demonstrate ECSE, the authors designed PET. Unlike many graph neural networks that focus on atoms (nodes), PET focuses on the connections between them (edges). It uses a transformer—the same tech behind GPT—to process these edge features. While PET isn't rotationally equivariant on its own, applying ECSE a posteriori (after training) makes it perfectly equivariant.

Datasets

The authors benchmarked their approach on a wide variety of challenging datasets:

  • Liquid Water: Configurations from path-integral molecular dynamics.
  • COLL: Highly distorted molecules undergoing collisions.
  • MnO: Manganese oxide crystals, including complex magnetic spin data.
  • HME21: A diverse set of high-temperature molecular dynamics configurations.
  • HEA (High-Entropy Alloys): Complex crystals containing up to 25 different transition metals.
  • Random CH4: Randomly displaced methane configurations.
  • QM9: A standard benchmark for molecular properties (specifically testing dipole moments).

Key Insights and Ideas

  • The "Design Space" Liberation: The most profound insight is that enforcing symmetry inside the model architecture might be holding us back. By moving the symmetry requirement to an external protocol (ECSE), researchers can use "off-the-shelf" innovations from the broader AI community.
  • Smoothness is Non-Negotiable: In physics, if you move an atom by a tiny amount, the energy should change by a tiny, predictable amount. The authors show that "plain" averaging of coordinate systems fails because it creates jumps. Their use of infinitely differentiable switching functions is what makes this practical for real-world simulations.
  • Edge-Centricity Matters: By focusing on edges rather than nodes, PET avoids "information bottlenecks" common in standard message-passing networks, allowing it to capture more complex geometric relationships.

Key Results

  • State-of-the-Art Performance: PET + ECSE outperformed existing "physics-native" models like NequIP and GemNet on several benchmarks.
  • Significant Error Reduction: On the liquid water dataset, PET reduced force errors by approximately 30% compared to the previous state-of-the-art.
  • Versatility: The model successfully handled not just energies and forces, but also vectorial properties (like dipole moments) and magnetic properties (collinear spins), which are often difficult for standard models to incorporate.
  • Minimal Accuracy Trade-off: The authors found that the difference in accuracy between the "raw" PET model and the symmetrized ECSE version was "minuscule," proving the protocol preserves the backbone's power.

Limitations and Future Work

The primary drawback is computational overhead. In its current proof-of-concept implementation, the ECSE protocol adds significant cost to inference—roughly three orders of magnitude slower than the base model. While the authors note that this is still much cheaper than the "first-principles" physics calculations they are replacing, it currently limits the model's use in massive, long-term simulations.

Future work will likely focus on optimizing this implementation. The authors suggest that "tightening" the ensemble (using fewer coordinate systems) or integrating the symmetry more deeply into the message-passing layers could drastically improve speed without losing the "smoothness" that makes the method so valuable.

Summary updated

Abstract

Point clouds are versatile representations of 3D objects and have found widespread application in science and engineering. Many successful deep-learning models have been proposed that use them as input. The domain of chemical and materials modeling is especially challenging because exact compliance with physical constraints is highly desirable for a model to be usable in practice. These constraints include smoothness and invariance with respect to translations, rotations, and permutations of identical atoms. If these requirements are not rigorously fulfilled, atomistic simulations might lead to absurd outcomes even if the model has excellent accuracy. Consequently, dedicated architectures, which achieve invariance by restricting their design space, have been developed. General-purpose point-cloud models are more varied but often disregard rotational symmetry. We propose a general symmetrization method that adds rotational equivariance to any given model while preserving all the other requirements. Our approach simplifies the development of better atomic-scale machine-learning schemes by relaxing the constraints on the design space and making it possible to incorporate ideas that proved effective in other domains. We demonstrate this idea by introducing the Point Edge Transformer (PET) architecture, which is not intrinsically equivariant but achieves state-of-the-art performance on several benchmark datasets of molecules and solids. A-posteriori application of our general protocol makes PET exactly equivariant, with minimal changes to its accuracy.
BibTeX
@misc{pozdnyakov2023point,
  author = {Sergey N. Pozdnyakov and Michele Ceriotti},
  title = {{Point Edge Transformer model trained on highly distorted methane configurations}},
  year = {2023},
  eprint = {2305.19302},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.19302},
}
Figure