Fine-tuning Vision Transformers for the Prediction of State Variables in Ising Models

Onur Kara, Arijit Sehanobish, Hector H. CorzoarXiv 2021 · arXiv

Introduction

The Ising model serves as a fundamental framework in statistical mechanics for understanding ferromagnetism and phase transitions. While the 2D Ising model is analytically solvable, more complex variations often require computationally intensive numerical methods. Recently, Machine Learning (ML) has emerged as a powerful tool for identifying phase transitions. Traditional approaches have largely relied on Convolutional Neural Networks (CNNs). However, CNNs possess inherent inductive biases, such as translation invariance, which may limit their ability to capture the long-range global correlations critical to physical systems near a phase transition. This study investigates the application of Vision Transformers (ViTs) to predict state variables in 2D Ising models, leveraging their attention mechanisms to overcome the limitations of local feature extraction.

Methodology and Dataset Generation

The researchers utilized a ViT-Base model, pre-trained on ImageNet-21k and fine-tuned on ImageNet-1k, followed by domain-specific fine-tuning on Ising model microstates. Unlike CNNs, which use sliding windows to process local features, ViTs decompose images into sequences of flattened patches. Self-attention mechanisms then allow the model to weigh the relative importance of these patches across the entire image, regardless of spatial distance.

To train and evaluate the models, a custom dataset of 100x100 lattice images was generated using the Metropolis-Hastings algorithm. The dataset included 1,300 images for each of three boundary conditions: Periodic, Anti-periodic, and Skewed ±. These microstates were categorized into four temperature-based bins representing different physical regimes:

Visual representation of the four temperature-based classification bins (A: Far sub-critical, B: Sub-critical, C: Critical Region, D: Super-critical).

Physical States and Phase Transitions

The model's task was to classify these microstates across various physical configurations, including both ferromagnetic and anti-ferromagnetic couplings. At low temperatures ($T=0$), ferromagnetic systems exhibit uniform alignment (all yellow or all blue), while anti-ferromagnetic systems form a perfect checkerboard pattern. As temperature increases ($T \gg T_c$), both systems transition into a disordered, high-entropy state.

Comparison of Anti-ferromagnetic and Ferromagnetic lattice configurations at ground state (T=0) and high temperature (T >> Tc).

Key Results and Performance

The performance of the ViT was compared against ResNet-18 and ResNet-50 baselines using Macro F1 scores. Remarkably, the ViT consistently outperformed the CNNs across all boundary conditions, even when trained on relatively small datasets. This suggests that the general features learned during ImageNet pre-training provide a robust foundation for physical state classification.

System/Boundary Condition ViT (F1) ResNet-50 (F1)
Periodic (Ferromagnetic) 0.934 0.907
Periodic (Anti-ferromagnetic) 0.935 0.899
Skewed (Ferromagnetic) 0.931 0.917
Anti-Periodic (Ferromagnetic) 0.921 0.917

The ViT demonstrated particular strength in the "Critical Region" (CR), a narrow temperature band characterized by high noise and subtle features, which is traditionally the most difficult region for classification.

Conclusion and Future Directions

This research highlights the potential of Vision Transformers in the field of computational physics. Beyond superior accuracy, ViTs offer enhanced interpretability through attention maps, which can reveal the specific lattice regions the model prioritizes during classification. Future work will extend this approach to more complex systems, such as the XY and q-Potts models, and explore the use of ViTs in discovering novel topological phase transitions.

Summary updated

Abstract

Transformers are state-of-the-art deep learning models that are composed of stacked attention and point-wise, fully connected layers designed for handling sequential data. Transformers are not only ubiquitous throughout Natural Language Processing (NLP), but, recently, they have inspired a new wave of Computer Vision (CV) applications research. In this work, a Vision Transformer (ViT) is applied to predict the state variables of 2-dimensional Ising model simulations. Our experiments show that ViT outperform state-of-the-art Convolutional Neural Networks (CNN) when using a small number of microstate images from the Ising model corresponding to various boundary conditions and temperatures. This work opens the possibility of applying ViT to other simulations, and raises interesting research directions on how attention maps can learn about the underlying physics governing different phenomena.
BibTeX
@misc{kara2021fine,
  author = {Onur Kara and Arijit Sehanobish and Hector H. Corzo},
  title = {{Fine-tuning Vision Transformers for the Prediction of State Variables in Ising Models}},
  year = {2021},
  eprint = {2109.13925},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2109.13925},
}
Figure