Repository of logically consistent real-world Boolean network models

Samuel Pastva, David Šafránek, Nikola Beneš, Luboš Brim, Thomas A. HenzingerbioRxiv 2023 · DOI

TL;DR

Researchers have created a comprehensive, open-source dataset of over 210 real-world Boolean network models, addressing the critical need for standardized, high-quality benchmarks in systems biology. This dataset includes a robust validation pipeline that identifies and corrects over 400 potential inconsistencies, making it a valuable resource for developing and comparing computational tools.

What Problem Does This Paper Address?

Systems biology increasingly relies on logical models, particularly Boolean networks (BNs), for computational analysis and automated reasoning. However, the field faces a significant challenge: a lack of standardized, high-quality benchmark models for testing and comparing new tools. Existing model repositories are often limited in scope, contain ad-hoc modifications, and lack rigorous validation, making it difficult to ensure the reliability and reproducibility of research. This hinders the advancement of automated reasoning techniques by providing an inconsistent foundation for evaluation.

Methods

The core of this work is the creation and curation of the Biodivine Boolean Models (BBM) dataset. The process involves:

  • Model Acquisition: Compiling models from various sources, including established databases (CellCollective, BioModels, GINsim) and literature surveys. The dataset accepts models based on real biological systems, regardless of their curation level or how they were generated (manual or automated).
  • Format Standardization: Supporting multiple common formats (bnet, aeon, sbml) and providing tools for conversion. Multi-valued networks are "Booleanized" using the Van Ham encoding for compatibility.
  • Automated Validation Pipeline: A crucial component that ensures the integrity and logical consistency of each model. This pipeline performs several checks:
  • Input Node Normalization: Standardizing how input variables (those with no incoming regulations) are represented.
  • Regulation Monotonicity and Essentiality: Verifying that the declared influence graph (which specifies regulatory relationships and their nature, e.g., activation/inhibition) is consistent with the actual update functions of the model. This is done using the symbolic BDD framework. Inconsistencies are repaired by adjusting the influence graph, and the changes are logged.
  • Unused Components: Ensuring that the network's influence graph is weakly connected, removing disconnected components to retain only the largest connected one.
  • Metadata Generation: Each model is accompanied by detailed metadata, including its source, publication, keywords, and information about any repairs made during validation.
  • Custom Dataset Export: A workflow allowing users to generate custom subsets of the dataset based on specific criteria, such as model format, input variable representation, or custom Python-based filtering.

Datasets

The paper introduces the Biodivine Boolean Models (BBM) dataset, a new, comprehensive, open-source collection.

  • Size: Contains over 210 real-world Boolean network models.
  • Sources: Compiled from existing databases like GINsim (83 models), CellCollective (78 models), BioModels (18 models), and the COVID-19 Disease Map (22 models), as well as literature surveys.
  • Key Differentiator: Unlike previous collections, BBM models undergo a rigorous validation pipeline to ensure logical consistency and integrity. The dataset also tracks model origins and modifications, providing transparency.
  • Formats: Models are available in bnet, aeon, and sbml formats.

Key Insights and Ideas

  • The Critical Need for Validated Benchmarks: The paper strongly emphasizes that the progress of computational systems biology hinges on reliable and reproducible evaluation. The BBM dataset directly addresses this by providing a large, well-defined, and validated collection of models.
  • Ubiquitous Inconsistencies in Existing Models: The validation pipeline uncovered a significant number of potential problems (over 400) in widely used models, highlighting the pervasive issue of logical inconsistencies between influence graphs and update functions. This underscores the importance of automated validation.
  • Subtle but Crucial Model Representations: The paper points out that seemingly minor differences in how input nodes or multi-valued variables are represented can have significant impacts on analysis results (e.g., attractor detection). The BBM dataset aims to standardize these representations.
  • Bridging the Gap Between Manual Curation and Automated Analysis: BBM facilitates the use of real-world biological models in automated analysis tools by providing them in standardized formats and ensuring their internal consistency, thereby reducing the burden on individual researchers.

Key Results

  • The BBM dataset comprises 210+ Boolean network models, making it one of the largest publicly available collections of real-world (non-randomly generated) models.
  • The validation pipeline identified and repaired over 400 potential problems across the models, with a significant number related to input monotonicity and essentiality mismatches.
  • Approximately 1.5% of all regulations were found to have potential integrity problems when compared to their corresponding update functions.
  • The dataset is distributed in multiple standard formats (bnet, aeon, sbml) and includes detailed metadata for each model.

Limitations and Future Work

  • Multi-valued Network Validation: The current validation pipeline cannot fully process multi-valued networks directly, as they are Booleanized. This means that certain types of logical inconsistencies specific to multi-valued representations might be missed.
  • Models Without Influence Graphs: For models that lack an explicit signed influence graph, the validation of regulation monotonicity and essentiality is limited. The authors plan to address this by comparing employed regulations against other models and biological databases in future work.
  • Potential for Uncovered Issues: The authors acknowledge that the validation process may have blind spots, and the total number of problems might not be final.
  • Future Work: The authors aim to address the limitations mentioned above, further expanding the dataset and improving the validation capabilities.

Summary updated

Abstract

ABSTRACT Recent developments in both computational analysis and data-driven synthesis enable a new era of automated reasoning with logical models (Boolean networks in particular) in systems biology. However, these advancements also motivate an increased focus on quality control and performance comparisons between tools. At the moment, to illustrate real-world applicability, authors typically test their approaches on small sets of manually curated models that are inherently limited in scope. This further complicates reuse and comparisons, because benchmark models often contain ad hoc modifications or are outright not available. In this paper, we describe a new, comprehensive, open source dataset of 210+ Boolean network models compiled from available databases and a literature survey. The models are available in a wide range of formats. Furthermore, the dataset is accompanied by a validation pipeline that ensures the integrity and logical consistency of each model. Using this pipeline, we identified and repaired 400+ potential problems in a number of widely used models.
BibTeX
@misc{pastva2023repository,
  author = {Samuel Pastva and David Šafránek and Nikola Beneš and Luboš Brim and Thomas A. Henzinger},
  title = {{Repository of logically consistent real-world Boolean network models}},
  year = {2023},
  doi = {10.1101/2023.06.12.544361},
  howpublished = {bioRxiv},
}

Loading related content...

Figure