When we train a Large Language Model (LLM) on text, a word like "cat" always represents a cat, regardless of where it appears in a sentence. In chemistry, however, the mapping between an object and its symbolic representation is far more fragile.
Molecules are inherently non-linear graphs—networks of atoms connected by bonds. To process them using sequence-based architectures like Transformers, chemists flatten these graphs into text strings, most commonly using the SMILES (Simplified Molecular Input Line Entry System) notation. The core issue is that a single molecular graph can be translated into dozens of different, equally valid SMILES strings depending on which atom a depth-first traversal algorithm chooses to start with.
This one-to-many mapping introduces a fundamental challenge: representation sensitivity. If a model cannot recognize that different textual sequences describe the exact same chemical structure, it lacks a true, representation-agnostic understanding of molecular identity.
Syntactic Memorization vs. Topological Understanding
To make string-based modeling manageable, the chemistry community typically relies on "canonicalization"—a standardized set of rules that selects a single, unique SMILES string for any given molecule. While this allows sequence-to-sequence models to achieve high accuracy on standard benchmarks, it creates a dangerous illusion of chemical intelligence.
As Kuzma Khrabrov et al. demonstrate in their work on the AMORE framework, chemical language models frequently memorize specific canonical string patterns rather than internalizing the invariant topological properties of the underlying graph. When these models are presented with non-canonical but entirely valid structural synonyms, their performance on downstream predictive tasks degrades sharply.
This failure of representation invariance indicates that the models are over-optimized for string syntax. Christoph Bartmann et al. exposed this brittleness using symbolically verifiable tasks in MolecularIQ, showing that even advanced reasoning models experience severe performance drops when input syntax is altered. Instead of evaluating the molecule itself, the models rely on string-based heuristics. As a result, they become over-reliant on notation syntax and fail to generalize across different molecular formats, such as shifting from SMILES to IUPAC names, as noted by Maksim Kuznetsov et al. in MMAI Gym for Science.
The Fragility of String Transformations
Why does a simple change in a text string cause a deep learning model's chemical reasoning to fall apart? The answer lies in how sequence models process token distributions.
Consider an "identity transformation"—a modification to a string that leaves the underlying chemical structure completely unchanged. A prime example is the transition between implicit and explicit data representations, such as explicitly writing out hydrogen atoms that are normally assumed to be there.
Canonical SMILES (Implicit H): CC(=O)O
Non-Canonical (Explicit H): [H]C([H])([H])C(=O)O[H]
To a human chemist, both strings describe acetic acid. But to a language model, the inclusion of explicit hydrogens completely shifts the token distribution. In a study on molecule captioning, Veronika Ganeeva et al. found that adding explicit hydrogens acts as a major disruptor, causing models to hallucinate incorrect structural features and output highly inconsistent natural language descriptions in Chemical Language Models Have Problems with Chemistry.
Furthermore, linear text strings struggle to capture global relationships in non-linear structures like rings and branches. The authors of tt DiffER point out that small local changes in a string representation can correspond to massive structural shifts in the actual molecule. Conversely, transformations like renumbering the atoms in a ring change the text completely while keeping the molecule identical. Veronika Ganeeva et al. showed in Lost in Translation that these minor modifications significantly degrade performance in both molecular retrieval and property prediction, leading to highly divergent and unreliable predictions for critical tasks like drug discovery.
Quantifying the Robustness Gap
To build better models, researchers have developed frameworks to systematically measure this sensitivity. The AMORE framework explicitly tests whether chemical language models map different string representations of the same molecule to identical internal embeddings. The findings reveal a near-total failure in representation consistency: the internal embeddings diverge wildly based on the input format.
This sensitivity is not confined to LLMs processing text. In Alchemical analysis of FDA approved drugs, Markus Orsi demonstrates that even traditional mathematical fingerprints create conflicting maps of chemical space depending on the specific formulation used. However, the problem is magnified in language models. Benchmarks like ChemIQ, introduced by Nicholas T. Runcie et al., confirm that model performance on molecular tasks is heavily dependent on standardized canonical formats, proving that a fully representation-agnostic understanding has not yet been achieved.
Pathways to Achieving Invariance
To overcome the limitations of string non-uniqueness, researchers are exploring three primary architectural strategies:
1. Permutation-Invariant Graph Models
The most direct way to achieve representation invariance is to abandon linear strings altogether and operate directly on molecular graphs. By using graph neural networks (GNNs) with permutation-invariant encoders, models can guarantee that the order in which reactants or atoms are listed does not alter the prediction.
For instance, Zhengkai Tu et al. developed Graph2SMILES, a graph-to-sequence model for retrosynthesis that eliminates the need for the massive data augmentation typically required to teach Transformers string invariance. Similarly, the authors of CGVAE generate molecules by building graphs one edge at a time, enforcing chemical validity directly within the graph structure rather than relying on fragile text syntax.
2. Syntax Realignment and Alternative Notations
If sequence-based models must be used, the underlying notation can be re-engineered to enforce tighter mappings. Jie Song et al. introduced R-SMILES (Root-aligned SMILES), which forces the string generation to start from the same "root" atom for both reactants and products in a chemical reaction. This creates a tight, one-to-one mapping that significantly reduces the computational burden on Transformer models. Other researchers, such as Kha-Dinh Luong et al., advocate for exploring more robust alternative string representations like SELFIES to mitigate the structural fragility inherent to SMILES.
3. Chemically Informative Latent Spaces
A third approach forces the model to map diverse string inputs into a shared, invariant latent space where structural synonyms naturally converge.
- Contrastive Learning: In LDMol, Jinho Chang et al. utilize contrastive learning to train a latent space where multiple valid SMILES strings for the same molecule are pulled together, allowing diffusion models to learn a consistent, chemically meaningful representation.
- Multimodal Fusion: Zihao Jing et al. developed MuMo, which fuses 2D topology, 3D geometry, and text sequences. By anchoring the model with a stable structural prior, it becomes robust not only to string variations but also to the geometric flexibility of 3D molecular conformers.
By moving away from superficial string matching and toward true topological and latent invariance, the next generation of chemical machine learning models aims to process molecules based on what they are, rather than how they are written.
Key Foundational Work
Constrained Graph Variational Autoencoders for Molecule Design (2018) — contributed 1 concept elements to this direction.
Inductive transfer learning for molecular activity prediction: Next-Gen QSAR Models with MolPMoFiT (2020) — contributed 1 concept elements to this direction.