Occupancies vs Embeddings: Internal representations of Ab Initio and Graph Probabilistic Models*

Incé A. Husain, Amer Marwan El Samman, Stijn De Baerdemacker

Abstract

Chemical modeling has traditionally been separated into two fairly non-overlapping frameworks: the exact framework of \textit{ab initio} models and the data-driven framework of statistical models. However, recent advances in machine learning approaches have intercepted the exact \textit{ab initio} frameworks by using an \textit{end-to-end} probabilistic model of the Schr"odinger equation in the Born-Oppenheimer approximation---from raw nuclear configurations to quantum observables such as the energy. While probabilistic modelling offers promising advancements to speed up computational predictions, the internal representation and its connection to the underlying quantum data---the wave function or electron density---which they implicitly rely on for training, remains relatively poorly understood. This study seeks to make a first comparison of the internal representation from a probabilistic end-to-end models with those from pure \textit{ab initio} frameworks such as density functional theory. We do this by comparing the internal atomwise representation of graph models, also called embeddings, to their most natural \textit{ab initio} counterpart, electron occupancy values. Our findings show that the embedding representations can be employed to transfer learn atomic occupancy values whereas the reverse mapping appears to be less accurate. We discuss the assumption that the probabilistic model could infer the existence of an underlying electron density underpinning the computational \textit{ab-initio} data.

Introduction

The field of computational chemistry is currently divided between two distinct paradigms. On one hand, ab initio methods like Density Functional Theory (DFT) provide high accuracy by solving fundamental physical equations, albeit at significant computational cost. On the other hand, "end-to-end" probabilistic models, such as Graph Neural Networks (GNNs), offer rapid predictions by learning directly from molecular data. However, these machine learning models are often viewed as "black boxes," with their internal representations—known as embeddings—poorly understood in relation to the physical principles they approximate.

This study investigates whether GNNs implicitly learn the underlying quantum mechanics of the systems they model. By comparing the internal atomwise embeddings of a SchNet model with physical electron occupancy values (Mulliken populations) derived from DFT, we explore the extent to which statistical models "understand" the electronic structure of molecules.

Methodology

To bridge the gap between statistical embeddings and physical occupancies, the researchers established a comparative framework using the QM9 dataset, which contains approximately 134,000 small organic molecules.

Figure 1: Conceptual framework for comparing GNN embeddings and DFT-calculated electron occupancies.

The methodology involved three primary components:

  1. Representation Generation: A SchNet GNN was pre-trained on 100,000 molecules to predict total energy, generating 128-dimensional embeddings for each atom. Simultaneously, DFT calculations provided Mulliken populations, representing the physical distribution of electrons in atomic orbitals.
  2. Dimensionality Analysis: Principal Component Analysis (PCA) and Linear Discriminant Analysis (LDA) were employed to visualize and classify the representations based on chemical functional groups.
  3. Transfer Learning: The core of the experiment involved bidirectional mapping. Small neural networks were trained to translate embeddings into occupancies ($E \to O$) and vice versa ($O \to E$) using a hold-out set of 4,750 molecules.

Results

The comparison revealed a significant asymmetry in the information content of the two representations.

Predictive Accuracy and Asymmetry

The mapping from GNN embeddings to physical occupancies ($E \to O$) was remarkably accurate, achieving an $R^2$ of 0.996 and a Mean Absolute Error (MAE) of $2.5 \times 10^{-3}$. This suggests that the GNN embeddings contain nearly all the information present in the physical occupancy data. Conversely, the reverse mapping ($O \to E$) performed poorly, with an $R^2$ of only 0.379. This asymmetry indicates that embeddings represent a "super-set" of information, capturing nuances of the chemical environment that simple occupancy values miss.

Functional Group Resolution

Using LDA, the GNN embeddings were able to classify atomic functional groups with near-zero error. In contrast, the physical occupancy data resulted in significantly higher error rates—13% for oxygen and 49% for hydrogen. For instance, embeddings could clearly distinguish between primary, secondary, and tertiary alcohols, whereas occupancy values for these atoms were nearly indistinguishable.

Impact on Energy Prediction

To test the necessity of this "extra" information, the researchers introduced noise into the embeddings to match the error levels of the occupancy data. This distortion caused energy predictions to fail significantly, with errors reaching 0.8 eV—well beyond the "chemical accuracy" threshold of 0.043 eV. This demonstrates that the high-resolution information captured by the GNN is essential for accurate energy modeling.

Discussion

The findings suggest that GNNs do not merely perform curve-fitting; they "discover" a representation of the underlying electron density. Even though the model was only trained on total energy, its internal architecture organized itself to reflect the physical distribution of electrons. The superior resolution of embeddings likely stems from their ability to encode the broader geometric context of an atom within a molecule, whereas Mulliken occupancies are more localized descriptors.

However, some limitations remain. The need to sum orbital data (e.g., $p_x, p_y, p_z$) to maintain rotational equivariance may discard some angular information. Additionally, the choice of Mulliken analysis as a partitioning scheme is one of several possible methods, and future studies could explore alternatives like Hirschfeld or Bader analysis.

Conclusion

This research provides a vital link between the "black box" of machine learning and the established principles of quantum chemistry. By demonstrating that GNN embeddings are a richer, more informative version of physical occupancies, the study suggests that probabilistic models are building a sophisticated internal language of chemistry. This "implicit physics" not only validates the use of GNNs in computational chemistry but also opens new avenues for using machine learning to gain deeper insights into molecular behavior.

Summary updated

Abstract

<title>Abstract</title> Chemical modeling has traditionally been separated into two fairly non-overlapping frameworks: the exact framework of \textit{ab initio} models and the data-driven framework of statistical models. However, recent advances in machine learning approaches have intercepted the exact \textit{ab initio} frameworks by using an \textit{end-to-end} probabilistic model of the Schr\"odinger equation in the Born-Oppenheimer approximation---from raw nuclear configurations to quantum observables such as the energy. While probabilistic modelling offers promising advancements to speed up computational predictions, the internal representation and its connection to the underlying quantum data---the wave function or electron density---which they implicitly rely on for training, remains relatively poorly understood. This study seeks to make a first comparison of the internal representation from a probabilistic end-to-end models with those from pure \textit{ab initio} frameworks such as density functional theory. We do this by comparing the internal atomwise representation of graph models, also called embeddings, to their most natural \textit{ab initio} counterpart, electron occupancy values. Our findings show that the embedding representations can be employed to transfer learn atomic occupancy values whereas the reverse mapping appears to be less accurate. We discuss the assumption that the probabilistic model could infer the existence of an underlying electron density underpinning the computational \textit{ab-initio} data.
BibTeX
@inproceedings{husain2026occupancies,
  author = {Incé A. Husain and Amer Marwan El Samman and Stijn De Baerdemacker},
  title = {{Occupancies vs Embeddings: Internal representations of Ab Initio and Graph Probabilistic Models*}},
  booktitle = {Theoretical},
  year = {2026},
  doi = {10.1007/s00214-025-03272-8},
}
Figure