Vision Transformers for X-ray Diffraction Patterns Analysis

Titouan Simonnet, Mame Diarra Fall, Sylvain Grangeon, Bruno GalerneICASSP 2025 · DOI

TL;DR

This paper introduces Vision Transformers (ViTs) for analyzing X-ray Diffraction (XRD) patterns, achieving state-of-the-art results in both mineral phase identification and proportion estimation. By adapting ViTs, the authors demonstrate superior generalization capabilities, particularly on real-world data, and offer novel visualization tools to understand the model's decision-making process.

What Problem Does This Paper Address?

Understanding the composition of materials is crucial in many scientific and industrial fields, and Powder X-ray Diffraction (XRD) is a powerful technique for this purpose. XRD patterns, which arise from how X-rays scatter off a material's crystalline structure, can be used to identify the mineral phases present (phase identification) and determine their relative amounts (phase quantification). Traditional methods like Rietveld refinement are effective but often require expert knowledge for initial phase identification, making them time-consuming or impractical for complex datasets like those from XRD computed tomography. While deep learning, particularly Convolutional Neural Networks (CNNs), has shown promise in analyzing XRD patterns, they face limitations due to the inherent structure of XRD signals. These signals are not invariant to translation and rely on the absolute positions and co-occurrence of multiple characteristic peaks, which can be challenging for standard CNN architectures. This paper aims to leverage the self-attention mechanism of Transformers, specifically Vision Transformers (ViTs), to overcome these limitations and improve both the accuracy and interpretability of XRD pattern analysis.

Methods

The core of this work is the adaptation of the Vision Transformer (ViT) architecture for 1D XRD signals. The process involves:

  • Input Embedding (Tokenization): The 1D XRD signal is divided into segments of a fixed length (C). This segmentation is a critical step, and the authors found C=80 to be optimal for their data. These segments are treated as "tokens."
  • Spectral Embedding & Positional Encoding: Each token is then embedded, and positional encoding vectors are added to retain information about the order of these segments.
  • Transformer Encoder: A standard Transformer encoder, consisting of multiple layers of Multi-Head Attention (MHA) and Multi-Layer Perceptrons (MLPs) with residual connections, processes these embedded tokens.
  • Task-Specific Heads: The output from the Transformer encoder, specifically the "class token" (CLS token), is fed into a linear head for either classification (phase identification) or proportion estimation.

A key innovation is the use of visualization tools to understand the ViT's behavior:

  • Positional Encoding Analysis: By examining the learnable positional encoding vectors, the authors can infer relationships and dependencies between different angular ranges in the XRD signal.
  • Attention Rollout: This technique visualizes how information propagates through the Transformer layers, highlighting which input tokens (segments of the XRD pattern) are most influential for the final prediction.

Datasets

The study utilizes synthetic XRD patterns for both tasks:

  • Phase Identification: Synthetic patterns for 6 mineral phases (Calcite, Halite, Hematite, Dolomite, Gibbsite, and Quartz) were used. The dataset comprised 1800 signals for training and 600 for testing.
  • Phase Quantification: A larger synthetic dataset was generated for mixtures of 4 mineral phases (Calcite, Dolomite, Gibbsite, and Hematite), with 10,000 patterns for training, 2,500 for validation, and 2,500 for testing.
  • Real-World Data: The quantification task was also evaluated on a set of 32 laboratory XRD patterns to assess generalization to real-world conditions.

Key Insights and Ideas

  • ViTs Excel at XRD's Non-Invariant Nature: The inherent structure of XRD patterns, where peak positions and their relationships are crucial, makes them well-suited for Transformer architectures that explicitly encode positional information and capture long-range dependencies via self-attention.
  • Tokenization is Crucial: The way the XRD signal is segmented into tokens significantly impacts ViT performance, highlighting the importance of careful preprocessing for 1D signal analysis with Transformers.
  • Visualization Unlocks Interpretability: Attention Rollout and positional encoding analysis provide unprecedented insights into why a ViT makes a particular prediction, bridging the gap between black-box deep learning models and scientific understanding. This is particularly valuable for complex scientific data.
  • Generalization Power: ViTs demonstrate superior generalization to real-world XRD data compared to CNNs, suggesting they are more robust to variations and noise present in experimental measurements.

Key Results

  • Phase Identification: A ViT variant (ViT-80) achieved an accuracy of 99.8% on synthetic data, outperforming a CNN which achieved 96%.
  • Phase Quantification (Synthetic Data): While CNNs performed better on synthetic data (RMSE ~0.49%, MMAE ~0.55%, RRS ~97.4%), ViTs showed higher errors (RMSE ~1.23%, MMAE ~1.56%, RRS ~87.2%).
  • Phase Quantification (Real Data): ViTs significantly outperformed CNNs on real laboratory data, achieving better results in MMAE (5.81% vs. 6.96%) and RRS (96.9% vs. 71.87%), indicating better generalization.
  • Computational Cost: ViT training required significantly more time (170 minutes) compared to CNNs (13 minutes) for quantification, but this was on 1000 epochs for ViT vs. 100 for CNN.

Limitations and Future Work

  • Computational Cost: The training time for ViTs, especially for quantification, is considerably longer than for CNNs.
  • Synthetic vs. Real Data Performance: While ViTs excel on real data, their performance on synthetic data for quantification is currently lower than CNNs.
  • Future Work: The authors suggest exploring scaling these results to larger and more diverse XRD databases, potentially developing a "foundation model" for XRD analysis. Further investigations into optimizing tokenization strategies and exploring different Transformer architectures for 1D signals are also promising avenues.

Summary updated

BibTeX
@inproceedings{simonnet2025vision,
  author = {Titouan Simonnet and Mame Diarra Fall and Sylvain Grangeon and Bruno Galerne},
  title = {{Vision Transformers for X-ray Diffraction Patterns Analysis}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2025},
  doi = {10.1109/icassp49660.2025.10887635},
}
Figure