Probabilistic Transformer: Modelling Ambiguities and Distributions for RNA Folding and Molecule Design

Jörg K. H. Franke, Frederic Runge, Frank Hutter

TL;DR

The ProbTransformer enhances the standard Transformer architecture by incorporating hierarchical latent distributions, allowing it to model "one-to-many" ambiguities in complex data. By sampling in the latent space rather than just the output, it achieves state-of-the-art results in RNA folding and property-based molecule design.

The Problem with Deterministic Models

Standard Transformers are essentially deterministic point-estimators. While incredibly powerful, they struggle with ambiguity. In many natural processes, a single input can correspond to multiple valid outputs.

Take RNA folding: a single sequence of nucleotides can fold into several different secondary structures within a cell. Similarly, in molecule design, many different chemical structures might satisfy the same set of target properties. A vanilla Transformer can only express uncertainty through its final softmax output. This often leads to "averaging" out the results or failing to capture the true underlying distribution of the data. To solve this, the authors aim to move probabilistic reasoning directly into the model's internal "latent" representations.

Methods: The Probabilistic Transformer

The core innovation of this paper is the Probabilistic Feed-Forward layer (prob layer). Instead of passing a fixed, deterministic vector between layers, the prob layer parameterizes a multivariate Gaussian distribution (with mean $\mu$ and variance $\sigma$).

The Probabilistic Feed-Forward layer replaces standard deterministic vectors with a parameterized Gaussian distribution.

Key technical components include:

  • Hierarchical Latent Distribution: These prob layers are inserted into multiple Transformer blocks. This allows the model to refine its "guess" about the data distribution hierarchically as the signal moves through the network.
  • GECO Training: To train this variational architecture, the authors use Generalized ELBO with Constrained Optimization (GECO). This treats the reconstruction loss as a constraint, helping the model balance learning the data versus regularizing the latent space.
  • Kappa Annealing: A novel contribution where the reconstruction constraint ($\kappa$) is adjusted online during training. This prevents the common "posterior collapse" issue in variational models, where the network learns to ignore the latent variables entirely.

Applications and Datasets

The authors evaluated the ProbTransformer across three distinct domains to test its ability to handle ambiguity:

  1. Synthetic Sequential Distribution Task: A new benchmark designed to test if a model can recover a hidden, sparse distribution of tokens.
  2. RNA Folding: Compiled from a large pool of publicly available RNA data (over 52,000 training samples). The authors introduced specific test subsets like TSsameStruc (different sequences, same structure) and TSsameSeq (one sequence, multiple structures) to rigorously test ambiguity handling.
  3. Molecule Design: Using the GuacaMol dataset, the model's ability to generate molecules conditioned on specific properties like solubility (logP) and surface area (TPSA) was evaluated.

Key Insights and Results

The ProbTransformer consistently outperformed vanilla Transformers and specialized baselines across all tested domains.

  • Latent vs. Softmax Sampling: The authors demonstrated that sampling from a learned latent distribution is far more expressive than simply sampling from the final softmax output. On the synthetic task, ProbTransformer achieved a KL-divergence of 0.52, vastly outperforming the vanilla Transformer's 7.84 (using softmax sampling).
  • Mastering "One-to-Many" Mappings: By keeping ambiguous data in the training set rather than filtering it out as "noise," the ProbTransformer learns that multiple answers can be correct. This leads to significantly better generalization on real-world biological data.
  • State-of-the-Art RNA Folding: On the standard TS0 test set, the model achieved an F1 score of 62.5 and solved 11.8% of structures—over eight times more than the next best deep learning model, MXFold2. It proved surprisingly adept at predicting pseudoknots, which are complex, non-nested RNA structures notoriously difficult for traditional algorithms to solve.
Visual comparison of RNA structure predictions. ProbTransformer accurately captures complex long-range interactions and pseudoknots compared to other models.
  • Molecule Design: In multi-property conditional generation, ProbTransformer outperformed MolGPT across almost all metrics. It was particularly effective in controlling the Topological Polar Surface Area (TPSA), reducing the Mean Average Deviation (MAD) by nearly 35%.

Limitations and Future Work

Despite its impressive capabilities, the ProbTransformer has a few drawbacks:

  • Computational Overhead: The addition of the posterior encoder and probabilistic layers increases memory and compute requirements by roughly a factor of two during training.
  • Architectural Constraints: The current training setup requires source and target sequences to be aligned, limiting its direct application to standard "encoder-decoder" tasks (like machine translation) without further modification.
  • Scaling: While successfully tested on biology and chemistry, the model hasn't yet been scaled to massive natural language processing (NLP) tasks. This remains a highly promising future direction for handling linguistic ambiguity.

Summary updated

Abstract

Our world is ambiguous and this is reflected in the data we use to train our algorithms. This is particularly true when we try to model natural processes where collected data is affected by noisy measurements and differences in measurement techniques. Sometimes, the process itself is ambiguous, such as in the case of RNA folding, where the same nucleotide sequence can fold into different structures. This suggests that a predictive model should have similar probabilistic characteristics to match the data it models. Therefore, we propose a hierarchical latent distribution to enhance one of the most successful deep learning models, the Transformer, to accommodate ambiguities and data distributions. We show the benefits of our approach (1) on a synthetic task that captures the ability to learn a hidden data distribution, (2) with state-of-the-art results in RNA folding that reveal advantages on highly ambiguous data, and (3) demonstrating its generative capabilities on property-based molecule design by implicitly learning the underlying distributions and outperforming existing work.
BibTeX
@inproceedings{franke2022probabilistic,
  author = {Jörg K. H. Franke and Frederic Runge and Frank Hutter},
  title = {{Probabilistic Transformer: Modelling Ambiguities and Distributions for RNA Folding and Molecule Design}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=QyCmjGYxbv},
}
Figure