Resolving the data ambiguity for periodic crystals

Daniel Widdowson, Vitaliy KurlinNeurIPS 2022 · OpenReview

Researchers have developed Pointwise Distance Distributions (PDD), a new "DNA-style" descriptor for periodic crystals that is both mathematically unique and computationally efficient. By resolving long-standing ambiguities in how we describe crystal structures, researchers successfully scanned the world’s largest database of materials, identifying numerous duplicates and fraudulent entries that had gone unnoticed for decades.

The Challenge of Data Ambiguity

In materials science, a crystal is essentially an infinite, repeating arrangement of atoms. However, describing this arrangement mathematically is surprisingly difficult. Traditionally, scientists use a "unit cell"—a small box that, when stacked, creates the whole crystal.

Different valid unit cells for the same periodic crystal structure.

The fundamental problem is that there are infinitely many ways to choose a unit cell for the exact same crystal. This "data ambiguity" means two researchers might describe the identical material using completely different coordinates, making it nearly impossible for computers to realize they are looking at the same thing.

Existing solutions are either too slow for large databases, "incomplete" (unable to tell some different structures apart), or "discontinuous" (where a tiny vibration in an atom’s position completely changes the descriptor). This has led to an "embarrassment of over-prediction" in simulations and thousands of undetected duplicates in experimental databases.

The Solution: Pointwise Distance Distributions (PDD)

To solve this, the authors introduce Pointwise Distance Distributions (PDD). Instead of relying on arbitrary bounding boxes or rigid symmetry groups, PDD looks at the local environment of every atom.

  1. Distance Matrices: For every atom in a repeating unit (motif), the algorithm calculates the distances to its $k$ nearest neighbors.
  2. Lexicographical Ordering: These distances are sorted to ensure the description doesn't change just because the neighbors were listed in a different order.
  3. Weighted Collapse: If multiple atoms have identical neighbor environments, they are collapsed into a single row with an assigned weight.
  4. Metric Comparison: To compare two different crystals, the authors use the Earth Mover’s Distance (EMD). This calculates the minimum "work" required to transform one distribution of distances into another.

Unlike previous methods, PDD is generic complete (it uniquely identifies almost any crystal) and Lipschitz continuous (small physical perturbations result in proportionately small changes to the PDD value).

Key Insights and Impact

  • The "DNA" of Materials: PDD acts as a unique fingerprint. Because it is "invertible," you can theoretically reconstruct the entire crystal structure just from its PDD invariant.
  • Symmetry is Fragile, Distances are Robust: Traditional crystallography relies on symmetry groups (like "cubic" or "hexagonal"). However, a tiny nudge to an atom can "break" symmetry mathematically while barely changing the material physically. PDD ignores these rigid labels in favor of continuous distance distributions, making it much more useful for noisy, real-world experimental data.

Unprecedented Scale and Speed

The authors evaluated PDD against the Cambridge Structural Database (CSD), which contains over 660,000 real-world periodic crystals. They performed over 200 billion pairwise comparisons of all periodic structures in the CSD.

This monumental task took only two days on a modest desktop computer. For context, the industry-standard "COMPACK" algorithm would have taken an estimated 34,000 years to complete the same comparisons.

Finding the Needles in the Haystack

Through this massive scan, PDD identified five pairs of unexpected duplicates in the CSD—structures that were supposedly different materials but were geometrically identical. These findings were so significant that five different scientific journals are currently investigating the data integrity of the original publications associated with those duplicates. The team also successfully tested PDD on a dataset of 5,679 predicted T2 molecule crystals to demonstrate how it handles "near-duplicates" in theoretical modeling.

Limitations and Future Work

While PDD is "generically complete," the authors acknowledge the hypothetical existence of "singular sets"—rare, highly specific mathematical arrangements where PDD might not be 100% unique, though none were found in real-world data.

Currently, the core PDD math treats atoms as unlabeled points, ignoring whether an atom is Carbon or Oxygen. While the authors discuss how to incorporate atomic types, the primary focus of this paper was on pure geometry. The next step is mapping the entire CRISP (Crystal Isometry Space)—a continuous "universe" of all possible crystal structures—to help researchers navigate and discover entirely new materials.

Summary updated

Abstract

The fundamental model of all solid crystalline materials is a periodic set of atomic centers considered up to rigid motion in Euclidean space. The major obstacle to materials discovery was highly ambiguous representations of periodic crystals that didn't allow fast and reliable comparisons and led to numerous (near-) duplicates in many databases of experimental and simulated crystals. This paper exemplarily resolves the ambiguity by invariants, which are descriptors without false negatives.The new Pointwise Distance Distributions (PDD) is a numerical matrix with a near-linear time complexity and an exactly computable metric. The strongest theoretical result is generic completeness (absence of false positives) for all finite and periodic sets of points in any dimension. The strength of PDD is shown by 200B+ pairwise comparisons of all periodic structures in the world's largest collection (Cambridge Structural Database) of existing materials over two days on a modest desktop.
BibTeX
@inproceedings{widdowson2022resolving,
  author = {Daniel Widdowson and Vitaliy Kurlin},
  title = {{Resolving the data ambiguity for periodic crystals}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=M5HOdElXv5},
}
Figure