Semi-Supervised Junction Tree Variational Autoencoder for Molecular Property Prediction

Atia Hamidizadeh, Tony Shen, Martin Ester

TL;DR

Researchers have developed SeMole, a semi-supervised generative model that predicts molecular properties and designs new molecules even when labeled data is scarce. By combining the structural reliability of Junction Tree Variational Autoencoders (JTVAE) with a semi-supervised framework, the model achieves 100% molecular validity and outperforms string-based methods in property prediction accuracy.

What Problem Does This Paper Address?

Drug discovery is a high-stakes game of "needle in a haystack." Scientists must navigate a vast chemical space to find small molecules with specific biological properties. While computational methods help, they face two major hurdles:

  1. Data Scarcity: Testing molecules in a "wet lab" to determine properties (like solubility or toxicity) is expensive and slow. Consequently, we often have millions of known molecular structures but very few labeled data points for specific properties.
  2. Structural Fragility: Many earlier models represent molecules as SMILES strings (text sequences). However, changing a single character in a SMILES string often results in a chemically impossible molecule.

Existing state-of-the-art models like the Junction Tree VAE (JTVAE) solve the validity problem by treating molecules as graphs and trees, but they are typically unsupervised—meaning they learn the "shape" of molecules without necessarily understanding their "function" (properties).

Methods

The authors introduce SeMole, which bridges the gap between graph-based generation and semi-supervised learning.

The Architecture

SeMole builds on the JTVAE by decomposing a molecular graph into a "junction tree" of chemically valid substructures (clusters). It introduces three distinct latent variables:

  • $z_T$: Captures the tree structure and the types of clusters used.
  • $z_G$: Captures how those clusters are connected in the final graph.
  • $y$: Represents the target molecular property.

By incorporating $y$ directly into the latent representation, the model can be trained to both predict properties for unlabeled molecules and generate new molecules conditioned on a specific desired property value.

The SeMole framework includes a generative model and two inference models to handle both labeled and unlabeled data.

The Pre-training "Trick"

Training semi-supervised VAEs is notoriously difficult because the model has to balance multiple stochastic variables at once. To stabilize this, the authors proposed SeMole$_{Pretrained}$. In this version, they initially set the supervised loss coefficient to zero, allowing the model to learn basic molecular structures first. They then gradually increase the importance of the property prediction task, which prevents the model from getting stuck in poor local optima early in training.

Datasets

The model was evaluated using the ZINC dataset, a standard benchmark in cheminformatics containing roughly 310,000 drug-like molecules. The researchers tested the model on three key chemical properties:

  • MolWt (Molecular Weight): The size of the molecule.
  • LogP (Partition Coefficient): A measure of how well a molecule dissolves in water vs. fats.
  • QED (Quantitative Estimation of Drug-likeness): A score indicating how likely a molecule is to be a viable drug.

Key Insights and Ideas

  • Graphs > Strings: The move from SMILES strings to graph-based building blocks is a game-changer for reliability. Because SeMole assembles molecules from a vocabulary of valid chemical components, it achieves 100% chemical validity in its generated outputs.
  • Leveraging the Unlabeled: The core "aha" moment is that even if we don't know the property of a molecule, its structure still tells us something about the distribution of molecules in chemical space. By training on both labeled and unlabeled data, SeMole builds a more robust latent space than a purely supervised model could.
  • Conditional Control: Unlike previous methods that required post-hoc optimization to find molecules with certain traits, SeMole allows researchers to simply "dial in" a property value in the latent space to generate matching candidates.

Key Results

The researchers compared SeMole against SSVAE (a semi-supervised model using SMILES) and a purely supervised version of their own model ($SeMole_{Supervised}$).

  • Superior Accuracy: As shown in Table 1, $SeMole_{Pretrained}$ consistently achieved lower Mean Absolute Error (MAE) across most properties, especially when labeled data was extremely limited (e.g., only 5% or 10% of the dataset).
  • Ablation Success: The $SeMole_{Pretrained}$ version outperformed the standard SeMole, proving that the gradual introduction of supervised loss is critical for performance.
  • Validity: While SMILES-based models often produce "gibberish" molecules, SeMole maintained perfect chemical validity throughout the generation process.
Table 1: Mean Absolute Error(MAE) of Molecule Property Prediction with Varying Percentage of Labeled data

Limitations and Future Work

While SeMole is a significant step forward, the authors note a few areas for growth:

  • Single-Property Focus: The current model is designed to generate molecules based on one property at a time. In the real world, drugs must satisfy multiple criteria simultaneously (e.g., high solubility and low toxicity).
  • 2D vs. 3D: The model currently operates on 2D graph representations. Moving to 3D representations could capture more nuanced spatial information that dictates how drugs bind to targets.
  • Scalability: While more scalable than some optimization-based approaches, the junction tree decomposition adds computational overhead compared to simpler sequence models.

Summary updated

Abstract

Molecular Representation Learning is essential to solving many drug discovery and computational chemistry problems. It is a challenging problem due to the complex structure of molecules and the vast chemical space. Graph representations of molecules are more expressive than traditional representations, such as molecular fingerprints. Therefore, they can improve the performance of machine learning models. We propose SeMole, a method that augments the Junction Tree Variational Autoencoders, a state-of-the-art generative model for molecular graphs, with semi-supervised learning. SeMole aims to improve the accuracy of molecular property prediction when having limited labeled data by exploiting unlabeled data. We enforce that the model generates molecular graphs conditioned on target properties by incorporating the property into the latent representation. We propose an additional pre-training phase to improve the training process for our semi-supervised generative model. We perform an experimental evaluation on the ZINC dataset using three different molecular properties and demonstrate the benefits of semi-supervision.
BibTeX
@misc{hamidizadeh2022semi,
  author = {Atia Hamidizadeh and Tony Shen and Martin Ester},
  title = {{Semi-Supervised Junction Tree Variational Autoencoder for Molecular Property Prediction}},
  year = {2022},
  eprint = {2208.05119},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2208.05119},
}
Figure