Data-Scarce, Multi-Granular, Multimodal: Benchmark Design for Biosynthetic Gene Clusters

Hanlin Xiao, Eriko Takano, Mauricio A. Álvarez, Rainer BreitlingarXiv 2026 · arXiv

The search for new antibiotics and anticancer agents often leads scientists to Biosynthetic Gene Clusters (BGCs)—groups of genes in bacteria, fungi, and plants that work together to produce complex, bioactive molecules. As deep learning moves into the "foundation model" era for biology, researchers are racing to build models that can "read" these genetic instructions. However, a major bottleneck has emerged: how do we actually know if these models are any good?

A new paper from researchers at the University of Manchester introduces BGC-Bench, a rigorous evaluation suite designed to move BGC modeling beyond simple classification and into the realm of real-world utility.

TL;DR

BGC-Bench is a comprehensive new benchmark for evaluating deep learning models of Biosynthetic Gene Clusters, featuring five diverse tasks across multiple biological scales and modalities. The study reveals a surprising "aha" moment: massive pretrained foundation models do not consistently outperform simple, parameter-free baselines, highlighting a significant gap in how current AI aligns genetic sequences with chemical reality.

What Problem Does This Paper Address?

While we have thousands of sequenced genomes, we have very little "ground truth" data for BGCs—only about 3,000 have been experimentally characterized. This data scarcity makes training and evaluating models incredibly difficult.

Existing benchmarks have largely relied on "coarse classification" (e.g., "Is this a polyketide cluster?"). This is like evaluating a translation AI by only asking it to identify if a text is in "a Romance language" or "a Germanic language." It doesn't tell you if the model actually understands the nuances of the "text" (the gene cluster) or if it can predict the "meaning" (the chemical product). Furthermore, existing evaluations often fail to test generalization—how a model performs when it encounters a completely new family of genes or a novel chemical structure.

Methods

The authors propose a framework based on two pillars: Applicability and Discriminability.

Figure 1: BGC-Bench evaluation design framework. Appli- cability is characterized along a generalization axis (random, phenotype-shift and genotype-shift splits) and a functionality axis (tasks from broad class prediction to BGC–molecule retrieval); performance across these settings is then used to assess discrim- inability, and active learning adds a limited-labeling-budget set- ting.

To instantiate this, they developed BGC-Bench, which evaluates 12 different encoders across three levels of biological granularity:

  1. Nucleotide level: Reading the raw DNA.
  2. Amino-acid level: Reading the protein sequences of individual genes.
  3. Pfam-domain level: Reading the functional "blocks" that make up the proteins.

The benchmark tests these models on five scientifically meaningful tasks:

  • A-domain substrate prediction: Predicting which building block a specific enzyme will grab.
  • MIBiG & NPClassifier prediction: Categorizing the cluster into broad and fine-grained chemical classes.
  • Chemical property regression: Predicting 13 physical properties (like solubility or drug-likeness) of the resulting molecule.
  • BGC-molecule retrieval: A multimodal task where the model must "match" a gene cluster to its correct chemical product from a pool of candidates.

Datasets

The core of the benchmark is built on MIBiG 4.0, the community-standard database for characterized BGCs. The authors introduced two critical types of "splits" to test generalization:

  • Genotype-shift: Holding out entire gene cluster families (GCFs).
  • Phenotype-shift: Holding out specific chemical scaffolds.

They also included an active learning setting, simulating a laboratory environment where a researcher has a limited budget to "label" (experimentally verify) new clusters.

Key Insights and Ideas

  • No Single Winner: The "best" way to represent a BGC depends on the task. Pfam-domain encoders lead on cluster-level tasks, while amino-acid encoders are superior for fine-grained domain-level predictions.
  • The "Baselines are Hard to Beat" Problem: In many cases, simple "Bag-of-Pfam" models (which just count the types of functional domains present) performed as well as, or better than, massive neural networks with millions of parameters.
  • The Alignment Gap: In the retrieval task, a simple combination of domain counts and traditional chemical fingerprints outperformed learned neural representations. This suggests that current models aren't yet capturing the complex "grammar" that links genetic sequences to chemical space.
  • Data is Still King: Active learning experiments showed that most tasks are still heavily data-limited. Interestingly, the ranking of which model is "best" changes depending on how much labeled data you have.

Key Results

The authors used a separability metric to show that BGC-Bench successfully distinguishes between models that previous benchmarks found identical.

Figure 1: BGC-Bench evaluation design framework. Appli- cability is characterized along a generalization axis (random, phenotype-shift and genotype-shift splits) and a functionality axis (tasks from broad class prediction to BGC–molecule retrieval); performance across these settings is then used to assess discrim- inability, and active learning adds a limited-labeling-budget set- ting.
  • Retrieval: The choice of BGC encoder accounted for 78–92% of the variance in performance, proving that the bottleneck in matching genes to molecules lies in how we represent the genes, not the chemicals.
  • Foundation Models: The largest model tested, Evo 2 7B, was actually among the weakest on several classification and retrieval tasks, reinforcing the idea that "bigger isn't always better" in specialized scientific domains.
  • Active Learning: Pretrained encoders showed a clear advantage at very small labeling budgets (e.g., 32 labels), but their lead often diminished as more data became available.

Limitations and Future Work

The authors acknowledge that MIBiG is still small, which limits the statistical power of the benchmark. Furthermore, BGC-Bench currently focuses on representation quality (how well the model "understands" the input) rather than generative tasks (like designing a new BGC from scratch) or detection tasks (finding BGCs in a messy genome).

Future iterations of the benchmark could expand to include other modalities, such as mass spectrometry data, providing an even more holistic view of the "DNA-to-Molecule" pipeline. For now, BGC-Bench stands as a vital "reality check" for the field, urging researchers to look beyond the hype of foundation models and focus on biologically grounded design.

Summary updated

Abstract

Biosynthetic gene clusters (BGCs) are co-located genes that encode the biosynthesis of specialized metabolites, an important source of antibiotics, anticancer agents, immunosuppressants, and agricultural bioactives. Deep learning, particularly self-supervised foundation models, has spurred interest in BGC sequence modeling, but evaluation has lagged behind: existing benchmarks often rely on coarse classification over small curated datasets, providing limited discrimination among models and insight into downstream utility. We propose a benchmark-design framework in which an informative benchmark should be discriminative, separating models with statistical confidence, and applicable, spanning scientifically meaningful tasks under different levels of generalization. We instantiate this framework as BGC-Bench, which derives five tasks from MIBiG, ranging from A-domain substrate prediction to BGC-molecule retrieval, evaluates them under genotype- and phenotype-shift splits, includes an active-learning setting for label-scarce discovery, and is easily extensible to new tasks and encoders. BGCs further provide a shared testbed across granularities and modalities: BGC-Bench compares twelve encoders on nucleotide, amino-acid, or Pfam-domain sequences, including compositional baselines for each input type, and combines BGC and chemical encoders for cross-modal retrieval. No single representation dominates: the best granularity depends on the biological scale of the task, and pretrained encoders do not consistently outperform other representations across tasks and splits. Active-learning experiments show that most tasks remain data-limited, encoder rankings change with labeling budget, and acquisition strategies yield modest gains. These results show that BGC model evaluation requires task-diverse, shift-aware benchmarks, and the framework can be generalized to benchmark design beyond BGCs.
BibTeX
@misc{xiao2026data,
  author = {Hanlin Xiao and Eriko Takano and Mauricio A. Álvarez and Rainer Breitling},
  title = {{Data-Scarce, Multi-Granular, Multimodal: Benchmark Design for Biosynthetic Gene Clusters}},
  year = {2026},
  eprint = {2611.02275},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.02275},
}
Figure