Quantitative Assessment of Crystal Structure Prediction

Sergio Rincón, Gabriel González, Nicolás Andrade, Rafael Velásquez, Sebastián Ojeda, Juanita Puentes, Nicolás Aparicio, Paula Cárdenas, Pablo Arbeláez

TL;DR

The authors introduce OPAL-Bench, a new, rigorous evaluation framework for Crystal Structure Prediction (CSP) that moves beyond simple geometric matching. Their findings reveal a critical "geometric-physical gap": models that appear accurate on paper often produce structures that are physically impossible, highlighting the need for evaluation protocols that account for both physical plausibility and the existence of multiple stable crystal forms (polymorphism).

What Problem Does This Paper Address?

Crystal Structure Prediction (CSP) is the "holy grail" of materials science—predicting how atoms will arrange themselves in 3D space given only a chemical formula. While deep generative models (like diffusion and flow matching) have accelerated this field, their evaluation has remained stuck in the past.

The authors identify four major flaws in current benchmarks:

  1. Threshold Sensitivity: Standard metrics like "Match Rate" depend on arbitrary tolerances. Change the threshold slightly, and the ranking of top models flips.
  2. The Polymorph Problem: Most benchmarks compare a prediction to a single "ground truth" reference. However, many compositions are polymorphic, meaning they can form multiple distinct, stable structures.
  3. Ignoring Physics: Current metrics focus on where atoms are (geometry) but ignore whether that arrangement makes sense energetically or physically (e.g., are atoms overlapping?).
  4. Unfair Comparisons: Some models are given "cheat sheets" (symmetry templates) at inference time. Comparing these "Symmetry-Informed" (SICSP) models directly to standard CSP models is an apples-to-oranges comparison.

Methods

The researchers developed the Orientation-invariant, Polymorph-Aware, Large Benchmark (OPAL-Bench) to fix these issues.

Figure 1: OPAL-Bench evaluation pipeline. OPAL-Bench evaluates seven CSP and two SICSP methods across four datasets, with four geometric and five physical metrics, treating CSP and SICSP as fundamentally distinct tasks. For each composition, a model predicts one structure, and we retrieve all known polymorphs, the distinct stable structures of the same composition, as candidate references. We then apply species-aware SE(3) alignment for geometric metrics, while physical metrics are SE(3)-invariant and require no alignment, and score each metric against its closest polymorph.

Key technical innovations include:

  • Species-Aware SE(3) Alignment: Before calculating distances, the benchmark uses an iterative procedure to align the predicted and reference structures in 3D space, ensuring that atoms of the same element are correctly matched.
  • Polymorph-Aware Scoring: Instead of one reference, the model is scored against the closest known stable polymorph for that composition.
  • Dual-Axis Metrics: It evaluates models on Geometric Distances (Chamfer, Hausdorff, Sinkhorn, and RMSE) and Physical Distances (Density, X-ray diffraction patterns, and Formation Energy calculated via Machine Learning Interatomic Potentials).
  • OPAL-CSP Baseline: The authors also released a new baseline model—an SE(3)-equivariant graph transformer trained with flow matching—to set a high bar for the standard CSP track.

Datasets

The benchmark evaluates models across four diverse datasets:

  • Perov-5: Structurally homogeneous perovskites.
  • Carbon-24: Pure carbon structures; notable for extreme polymorphism (many ways to arrange carbon atoms).
  • MP-20 & MPTS-52: Large, diverse sets of inorganic crystals from the Materials Project, containing up to 20 and 52 atoms per cell, respectively.

Key Insights and Ideas

  • The "Aha!" Moment: Geometry $\neq$ Physics. The most striking finding is that geometric accuracy is weakly correlated with physical plausibility. A model might place atoms near their correct positions (low RMSE) but accidentally overlap them, leading to astronomical formation energies.
  • Flow vs. Diffusion: Flow-matching models (like the new OPAL-CSP and FlowMM) generally lead on geometric metrics, but diffusion-based models (like DiffCSP) are often more competitive on physical metrics like density.
  • Symmetry is a Massive Constraint: When models are given symmetry templates (SICSP), their performance skyrockets (see Figure 8 in the paper). This confirms that SICSP and standard CSP must be treated as separate tasks to maintain a fair leaderboard.

Key Results

The authors benchmarked seven state-of-the-art CSP models and two SICSP models.

Table 1: CSP Results. This table shows the median scores across four datasets. Note that no single method dominates all metrics, and some models with good geometric scores (like FlowLLM on MPTS-52) fail spectacularly on physical metrics like Formation Energy.
  • No Universal Winner: No single model dominates across all datasets and metrics. CSP remains an open, challenging problem.
  • OPAL-CSP Performance: The authors' baseline, OPAL-CSP, achieved the best geometric scores on Perov-5 and MP-20 and the best physical scores on MPTS-52, proving that a single model can indeed balance both worlds.
  • Failure Modes: The benchmark successfully exposed "geometric artifacts." For example, FlowLLM achieved competitive geometric scores but produced unphysical "superposed" atoms, resulting in a median formation energy error of 3264 eV/atom on MPTS-52—orders of magnitude higher than its peers.

Limitations and Future Work

While OPAL-Bench is a significant step forward, the authors acknowledge several areas for growth:

  • Reference Bias: The benchmark is limited by the diversity of polymorphs currently in the reference datasets. As crystal databases grow, the benchmark will become even more robust.
  • Diversity Metrics: Currently, the benchmark evaluates a single prediction per composition. Future iterations could measure "generative diversity"—how many different valid polymorphs a model can discover for a single formula.
  • MLIP Dependence: Physical metrics like formation energy depend on the accuracy of the underlying Machine Learning Interatomic Potentials (MLIPs). The authors used an ensemble to mitigate this, but better potentials will lead to better evaluation.

Summary updated

Abstract

Crystal Structure Prediction (CSP) aims to predict a material's 3D structure from its chemical composition, with applications from energy storage to biomedical devices. Despite its importance, existing evaluation frameworks suffer from key limitations: they rely on threshold-sensitive or incomparable metrics, focus predominantly on geometric performance while overlooking physical plausibility, and fail to account for polymorphism, where a chemical composition can crystallize into multiple stable structures. In parallel, recent Symmetry-Informed CSP (SICSP) methods often report results alongside standard CSP baselines while receiving crystallographic templates at inference, which constrain the search space and make SICSP a fundamentally different task. To address these limitations, we introduce an Orientation-invariant, Polymorph-Aware, Large Benchmark (OPAL-Bench) that evaluates CSP and SICSP as separate tasks. We evaluate 7 CSP and 2 SICSP methods under OPAL-Bench across 4 datasets using threshold-free geometric and physical distance metrics. Our results show that no method dominates across all metrics, and that geometric performance is not strongly correlated with physical plausibility. We also release OPAL-CSP, an SE(3)-equivariant flow-matching baseline for the CSP track that performs competitively on geometric metrics and ranks among the best on physical ones. By unifying geometric and physical metrics under a polymorph-aware protocol, OPAL-Bench enables a more comprehensive assessment of generative quality in CSP.
BibTeX
@misc{rincon2026quantitative,
  author = {Sergio Rincón and Gabriel González and Nicolás Andrade and Rafael Velásquez and Sebastián Ojeda and Juanita Puentes and Nicolás Aparicio and Paula Cárdenas and Pablo Arbeláez},
  title = {{Quantitative Assessment of Crystal Structure Prediction}},
  year = {2026},
  eprint = {2611.10380},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.10380},
}
Figure