Defining the Applicability Domain of Chemical Machine Learning

37 concepts26 papers1 proposal

Machine learning models have become remarkably adept at predicting molecular properties, from the taste profiles of novel compounds to the toxicity of environmental pollutants. However, chemical space is practically infinite, while training datasets are inherently sparse. When a model is asked to make a prediction for a molecule that is structurally or statistically distinct from anything it has seen before, it does not gracefully admit ignorance. Instead, it often yields highly confident, wildly inaccurate extrapolations.

To prevent these silent failures, researchers use a mechanism known as the Applicability Domain (AD). An applicability domain is a theoretical and mathematical boundary drawn around a model’s training data. It defines the region of chemical space where the model’s predictions can be trusted. Rather than improving the model's accuracy directly, an AD acts as a quality-control filter, separating trustworthy inferences from untrustworthy extrapolations.

The Problem of Optimization Blind Spots

Why do we need an explicit boundary? Why not simply rely on the model's internal confidence or average test-set error?

The danger becomes apparent during high-throughput screening or generative molecular design. As noted by Rittig et al. in Graph Machine Learning for Design of High-Octane Fuels, optimization algorithms are designed to maximize a target property. If a predictive model has an unconstrained extrapolation error in an uncharted region of chemical space, the optimization algorithm will inevitably exploit that blind spot. It will propose unrealistic or physically impossible molecules because the model mistakenly predicts they have exceptional performance.

This issue is not unique to fuel design. In a review of materials science models, Duan et al. in Machine Learning for the Discovery, Design, and Engineering of Materials emphasize that without robust uncertainty quantification, it is incredibly difficult to determine whether a model's prediction is a valid inference or an unreliable extrapolation beyond its learned domain. As Sutton argues in Identifying domains of applicability of machine learning models for materials science, relying on average test errors can be misleading; a model that appears mediocre on average might be exceptionally accurate within a specific structural subgroup, while a globally "accurate" model might fail catastrophically on a novel crystal composition.

How to Draw the Boundary

Defining an applicability domain requires translating the intuitive concept of "chemical similarity" into a rigorous mathematical constraint. Researchers achieve this through several distinct methodologies:

Distance-Based Metrics

The most common way to define an AD is to calculate how far a query molecule sits from the distribution of the training data. In Towards the prediction of drug solubility in binary solvent mixtures at various temperatures using machine learning, Bao et al. utilize metrics like the Mahalanobis distance. Unlike simple Euclidean distance, the Mahalanobis distance accounts for the correlations between different chemical descriptors, identifying when a test molecule violates the statistical distribution of the training set.

Other distance-based approaches include:

  • Standard Deviation Distance to Model: Used by Lei et al. in [ADMET evaluation in drug discovery:
  • Accurate prediction of rat oral acute toxicity using relevance vector machine and consensus modeling](/work/admet-evaluation-in-drug-discovery-15-accurate-prediction) to establish a quantitative threshold for toxicity models.
  • Leverage and k-Nearest Neighbors (k-NN): As highlighted by Fan et al. in QSAR ligand dataset for modelling mutagenicity, genotoxicity, and rodent carcinogenicity, leverage metrics assess how much influence a query point has on the model's space, while k-NN distances measure the local density of training samples around the query.

One-Class Classification and Density Estimation

Instead of calculating raw distances, some frameworks train a secondary model whose sole job is to recognize the boundaries of the training data. Rittig et al. implement one-class classification to actively penalize candidate molecules that stray too far from known chemical regimes. Similarly, Sánchez Medina et al. in Gibbs–Helmholtz graph neural network: capturing the temperature dependency of activity coefficients at infinite dilution show that reliability can be quantified by combining molecular similarity thresholds with training set density, ensuring a chemical class has sufficient representation before trusting a prediction.

Geometric and Conformational Thresholds

In deep learning models that operate on 3D molecular structures, such as interatomic potentials, the boundary must also account for geometric distortions. Shenoy et al. demonstrate in Role of Structural and Conformational Diversity for Machine Learning Potentials that prediction errors can increase by an order of magnitude when the geometric distance between a test conformer and the training data crosses a critical threshold, underscoring the need for strict spatial AD boundaries.

  Chemical Space
  ┌────────────────────────────────────────────────────────┐
  │                                                        │
  │        ▲ Training Data Cluster                         │
  │     ▲  ▲  ▲                                            │
  │    ▲  ▲  ▲  ▲      ┌────────────────────────┐          │
  │     ▲  ▲  ▲        │  Applicability Domain  │          │
  │                    └────────────────────────┘          │
  │                                                        │
  │                                     ○ Query Molecule B │
  │                                       (Out-of-Domain:  │
  │   ○ Query Molecule A                   High Uncertainty│
  │     (In-Domain:                        Rejected)       │
  │      Reliable Prediction)                              │
  │                                                        │
  └────────────────────────────────────────────────────────┘

The Trade-Off Between Scope and Accuracy

Implementing an applicability domain introduces a fundamental engineering trade-off: prediction accuracy versus model utility.

When you enforce a strict AD filter, you discard high-uncertainty predictions. As a result, the accuracy of the remaining predictions increases dramatically. For example, Fan et al. demonstrate in Comparison of seven in silico tools for evaluating of daphnia and fish acute toxicity: case study on Chinese Priority Controlled Chemicals and new chemicals that computational toxicology tools become significantly more reliable and trustworthy when predictions are strictly restricted to their defined AD.

However, this accuracy comes at a cost. By restricting the model to a narrow domain, you reduce the number of compounds you can evaluate. As Proia et al. observe in Ligand-based and structure-based studies to develop predictive models for SARS-CoV-2 main protease inhibitors through the 3d-qsar.com portal, while an AD filter improves model performance, it simultaneously limits the scope of novel chemical scaffolds the model can confidently explore.

Applications in Pipeline Automation

Despite this limitation, defining an AD is essential for automating discovery pipelines. In industrial settings, filtering out-of-distribution samples allows researchers to safely delegate low-risk decisions to AI while flagging anomalous molecules for expensive laboratory validation.

In drug discovery pipelines, such as the Caco-2 permeability framework designed by Wang et al. in [ADMET evaluation in drug discovery:

  1. Application and industrial validation of machine learning algorithms for Caco-2 permeability prediction](/work/admet-evaluation-in-drug-discovery-21-application), the AD acts as an automated gatekeeper. Molecules falling outside the reliable training space are treated as high-uncertainty samples and filtered out.

By embedding these boundaries directly into generative models—such as the target-focused library generator developed by Rybka et al. in PROFIS: Design of Target-Focused Libraries by Probing Continuous Fingerprint Space with Recurrent Neural Networks or the multi-task screening pipeline by Zhang et al. in Discovery of novel PRMT1 inhibitors: a combined approach using AI classification model and traditional virtual screening—researchers ensure that AI-generated leads remain grounded in realistic chemical regimes, drastically lowering false-positive rates in the lab.

Key Foundational Work

Estimation of acute oral toxicity in rat using local lazy learning (2014) — contributed 3 concept elements to this direction.
Applicability Domains for Classification Problems: Benchmarking of Distance to Models for Ames Mutagenicity Set (2010) — contributed 3 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_applicability_domain_definition_ca5c1244,
  title = {applicability domain definition},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/applicability-domain-definition/rdr_ca5c1244a71b4c5bb5c7de40c8f9ce45},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_applicability_domain_definition_ca5c1244_v1,
  title = {applicability domain definition},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/applicability-domain-definition/rdr_ca5c1244a71b4c5bb5c7de40c8f9ce45/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake