HMAE: Self-Supervised Few-Shot Learning for Quantum Spin Systems

Ibne Farabi Shihab, Sanjeda Akter, Anuj Sharma

TL;DR

Quantum machine learning struggles with scarce labeled data. This paper introduces HMAE, a self-supervised learning framework that pre-trains transformers on unlabeled quantum Hamiltonians. HMAE achieves impressive few-shot learning performance on quantum tasks by using a physics-informed masking strategy, significantly reducing the need for labeled data.

What Problem Does This Paper Address?

Simulating quantum systems, like spin lattices and molecules, is crucial for materials science and quantum chemistry. However, these simulations are computationally expensive, leading to a scarcity of labeled data for training machine learning models. Existing quantum machine learning approaches often require substantial labeled datasets and struggle with transferring knowledge between different quantum systems or tasks. This paper tackles the challenge of data scarcity and poor transferability in quantum machine learning by developing a self-supervised learning framework that can learn effectively from unlabeled quantum Hamiltonians.

Methods

The core of this paper is the Hamiltonian-Masked Autoencoding (HMAE) framework. It leverages a transformer architecture (specifically, a QuantumFormer encoder-decoder) for learning representations of quantum Hamiltonians. The key innovation lies in its physics-informed masking strategy. Instead of randomly masking parts of the Hamiltonian (like in standard masked autoencoders), HMAE selectively masks terms based on their physical significance, informed by quantum information theory. This involves calculating a "saliency score" for each Hamiltonian term, considering its energy contribution (coefficient magnitude) and its quantum mechanical interaction with other terms (commutator norms and site overlap).

The pre-training objective involves reconstructing the masked Hamiltonian terms. After pre-training, the encoder part of the QuantumFormer is used as a feature extractor for downstream tasks like phase classification and ground state energy prediction. This allows for few-shot learning, where the model can be fine-tuned with a very small number of labeled examples.

Datasets

The paper uses a comprehensive dataset of 12,500 quantum Hamiltonians for pre-training. This dataset consists of:

  • 60% real-world Hamiltonians (from sources like the Materials Project, QM9-X, PubChemQC, and quantum processors).
  • 40% synthetic Hamiltonians (Heisenberg, Ising, XY models, molecular systems, and theoretical models).

For experimental validation, they use a test set of 1,250 real quantum systems from various platforms (Harvard, MIT, IBM Quantum, trapped-ion experiments). They also created an additional test set of 350 larger quantum systems (16-20 qubits) to evaluate scalability.

Key Insights and Ideas

  • Physics-informed masking is crucial: Simply masking random parts of a Hamiltonian is less effective than masking based on physical significance. Prioritizing terms that are energetically important and interact strongly leads to better representations.
  • Self-supervision can overcome data scarcity: By learning from unlabeled Hamiltonians, HMAE significantly reduces the need for expensive, labeled quantum data, making quantum machine learning more accessible.
  • Transformers are well-suited for quantum Hamiltonians: With appropriate tokenization and masking, transformer architectures can effectively capture the complex structure of quantum Hamiltonians.
  • Transfer learning is powerful across quantum systems: HMAE demonstrates strong generalization capabilities, allowing models pre-trained on one set of quantum systems to perform well on others, even with different hardware or system sizes.
  • Quantum information theory provides a principled basis: The saliency score and masking strategy are grounded in quantum mutual information, offering a theoretically sound approach to feature selection.

Key Results

  • HMAE achieves 85.3% ± 1.5% accuracy in phase classification and 0.15 ± 0.02 eV MAE in ground state energy prediction with only 10 labeled examples.
  • This represents a 3-5x reduction in required labeled examples compared to baseline methods.
  • HMAE outperforms classical graph neural networks (GNNs) and quantum neural networks (QNNs) significantly, with p < 0.01 statistical significance.
  • The model shows robust performance on larger quantum systems (up to 20 qubits) and demonstrates strong transfer learning capabilities across different system sizes and experimental platforms.
  • The physics-informed masking strategy preserves more quantum information compared to energy-only or random masking.

Limitations and Future Work

  • Scalability: The current approach is limited to small quantum systems (up to 12 qubits for training, with extensions to 16-20 qubits for testing). Scaling to larger systems faces combinatorial growth in token complexity.
  • Ground truth requirement: While HMAE reduces the number of labeled examples needed for fine-tuning, obtaining high-quality ground truth values (from exact diagonalization or tensor networks) for these few examples is still necessary.
  • Theoretical limitations: The theoretical framework has limitations regarding high-entanglement systems and requires approximations.
  • Future Work: The authors suggest exploring sparse attention mechanisms, tensor network decoders, and continuous representation learning for better scalability. They also propose extending the framework to handle time-dependent Hamiltonians, multi-modal quantum learning, and quantum federated learning. Further research into theoretical foundations and applications in quantum error mitigation, materials discovery, and quantum control is also envisioned.

Summary updated

Abstract

Quantum machine learning for spin and molecular systems faces critical challenges of scarce labeled data and computationally expensive simulations. To address these limitations, we introduce Hamiltonian-Masked Autoencoding (HMAE), a novel self-supervised framework that pre-trains transformers on unlabeled quantum Hamiltonians, enabling efficient few-shot transfer learning. Unlike random masking approaches, HMAE employs a physics-informed strategy based on quantum information theory to selectively mask Hamiltonian terms based on their physical significance. Experiments on 12,500 quantum Hamiltonians (60% real-world, 40% synthetic) demonstrate that HMAE achieves 85.3% $\pm$ 1.5% accuracy in phase classification and 0.15 $\pm$ 0.02 eV MAE in ground state energy prediction with merely 10 labeled examples - a statistically significant improvement (p < 0.01) over classical graph neural networks (78.1% $\pm$ 2.1%) and quantum neural networks (76.8% $\pm$ 2.3%). Our method's primary advantage is exceptional sample efficiency - reducing required labeled examples by 3-5x compared to baseline methods - though we emphasize that ground truth values for fine-tuning and evaluation still require exact diagonalization or tensor networks. We explicitly acknowledge that our current approach is limited to small quantum systems (specifically limited to 12 qubits during training, with limited extension to 16-20 qubits in testing) and that, while promising within this regime, this size restriction prevents immediate application to larger systems of practical interest in materials science and quantum chemistry.
BibTeX
@inproceedings{shihab2025hmae,
  author = {Ibne Farabi Shihab and Sanjeda Akter and Anuj Sharma},
  title = {{HMAE: Self-Supervised Few-Shot Learning for Quantum Spin Systems}},
  booktitle = {ECAI},
  year = {2025},
  doi = {10.3233/faia251031},
}
Figure