Non-autoregressive electron flow generation for reaction prediction

Jian Tang (唐建)arXiv 2020 · arXiv

TL;DR

This paper introduces the Non-autoregressive Electron Flow Generator (NEFG), a novel deep learning model for predicting chemical reactions. By reframing reaction prediction as parallel electron flow generation rather than sequential token prediction, NEFG achieves state-of-the-art top-1 accuracy with a significant reduction in inference latency, making it a more efficient tool for chemical synthesis planning.

What Problem Does This Paper Address?

Predicting the outcome of chemical reactions is a fundamental challenge in computational chemistry, crucial for tasks like synthesis planning and drug discovery. Existing machine learning approaches often treat this as a sequence-to-sequence problem, similar to machine translation. However, these methods are typically autoregressive, meaning they generate the reaction step-by-step, conditioning each step on the previous ones. This sequential nature leads to high inference latency, as it can take hundreds of steps to predict a single reaction. While some graph-based methods reduce latency by predicting graph edits, they often sacrifice accuracy and rely on specific ordering assumptions. This paper aims to bridge this gap by developing a non-autoregressive model that predicts reactions in parallel, achieving both high accuracy and speed.

Methods

The core innovation of NEFG lies in its non-autoregressive approach, inspired by chemical principles. Instead of predicting tokens or graph edits sequentially, NEFG models electron flow. It represents a molecule as a graph where atoms are nodes and bonds are edges representing shared electrons. A chemical reaction is then viewed as a redistribution of these electrons, or "electron flows," between atoms.

The NEFG architecture consists of:

  • Encoder: A bi-level encoder using a Graph Convolutional Network (GCN) for local feature extraction and a Graph Attention Network (GAT) for capturing global graph structure. This encoder processes the reactant molecules.
  • Variational Encoder: This component estimates a latent variable z that captures the uncertainty in reaction outcomes, enabling the generation of diverse, multi-modal predictions.
  • Decoder: A non-autoregressive decoder that takes the encoded reactant information and the latent variable z to predict the electron flows (edge transformations) in parallel. This is achieved using a multi-pointer network that predicts incoming (influx) and outgoing (outflux) electron movements for each atom.

The model is trained using a loss function that combines reconstruction loss (ensuring the predicted product matches the ground truth) and a KL-divergence term (encouraging diversity in predictions, controlled by a $\beta$ hyperparameter).

Datasets

The primary dataset used for training and evaluation is the USPTO-MIT dataset. This dataset contains approximately 480,000 organic reactions, with reactants, reagents, and products represented as SMILES strings. The paper notes that this dataset has been widely used in previous reaction prediction research.

Key Insights and Ideas

  • Electron Flow as Parallelizable Units: The key insight is that electron redistribution in a chemical reaction can be conceptualized as parallel "electron flows" between atoms, rather than a strict sequential process. This allows for non-autoregressive prediction.
  • Bridging Chemistry and Graph Theory: The paper effectively connects chemical concepts like electron sharing and bond formation/breaking to graph theory concepts like nodes, edges, and edge transformations.
  • Latent Variables for Diversity: Incorporating latent variables (inspired by Variational Autoencoders) is crucial for capturing the inherent stochasticity of chemical reactions and generating multiple plausible product outcomes.
  • Non-Autoregressive Advantage: The non-autoregressive nature directly addresses the high inference latency of previous state-of-the-art models, offering a significant speedup without compromising accuracy.

Key Results

  • State-of-the-Art Top-1 Accuracy: NEFG achieves the highest top-1 accuracy among the evaluated methods on the USPTO-MIT dataset.
  • Significant Speedup: The model demonstrates an order of magnitude lower inference latency compared to autoregressive models like the Molecule Transformer, achieving a 26x speedup in one comparison.
  • Comparable Top-K Performance: While excelling in top-1 accuracy, NEFG's top-k accuracy is comparable to existing state-of-the-art methods, especially when enhanced with latent variables for multimodal output generation.
  • Robustness Across Reaction Topologies: The model performs well across different reaction topologies (single-linear, parallel linear, and non-linear), indicating its generalizability.

Limitations and Future Work

  • Heuristic Thresholds for Edge Prediction: The paper mentions that the rounding of continuous weights to integer edge modifications uses heuristically chosen thresholds. This could be an area for improvement.
  • Data Augmentation for Autoregressive Models: The paper highlights that autoregressive models like the Transformer often require data augmentation (e.g., random ordering of outputs) to achieve state-of-the-art performance. NEFG's parallel nature bypasses this need, but it's a point of comparison.
  • Future Work: The authors suggest that their framework is a promising new tool for designing non-autoregressive algorithms for graph generation, implying potential applications beyond reaction prediction. The ability to generate diverse multimodal outputs also opens avenues for exploring reaction pathways and exploring chemical space more broadly.

Summary updated

Abstract

Reaction prediction is a fundamental problem in computational chemistry. Existing approaches typically generate a chemical reaction by sampling tokens or graph edits sequentially, conditioning on previously generated outputs. These autoregressive generating methods impose an arbitrary ordering of outputs and prevent parallel decoding during inference. We devise a novel decoder that avoids such sequential generating and predicts the reaction in a Non-Autoregressive manner. Inspired by physical-chemistry insights, we represent edge edits in a molecule graph as electron flows, which can then be predicted in parallel. To capture the uncertainty of reactions, we introduce latent variables to generate multi-modal outputs. Following previous works, we evaluate our model on USPTO MIT dataset. Our model achieves both an order of magnitude lower inference latency, with state-of-the-art top-1 accuracy and comparable performance on Top-K sampling.
BibTeX
@misc{2020non,
  author = {Jian Tang (唐建)},
  title = {{Non-autoregressive electron flow generation for reaction prediction}},
  year = {2020},
  eprint = {2012.12124},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2012.12124},
}
Figure