OBO Foundry in 2021: Operationalizing Open Data Principles to Evaluate Ontologies

Rebecca Jackson, Nicolas Matentzoglu, James A. Overton, Randi Vita, James P. Balhoff, Pier Luigi Buttigieg, Seth Carbon, Mélanie Courtot, Alexander D. Diehl, Damion Dooley, William D. Duncan, Nomi L. Harris, Melissa Haendel, Suzanna Lewis, Darren A. Natale, David Osumi-Sutherland, Alan Ruttenberg, Lynn M. Schriml, Barry Smith, Christian J. Stoeckert, Nicole Vasilevsky, Ramona Walls, Jie Zheng, Chris Mungall, Bjoern PetersbioRxiv 2021 · DOI

TL;DR

This paper tackles the challenge of integrating diverse biological ontologies by operationalizing the Open Biological and Biomedical Ontologies (OBO) Foundry's guiding principles into automated checks. They've developed a dashboard that objectively evaluates ontologies against these principles, aiming to improve their quality, interoperability, and alignment with FAIR data standards.

What Problem Does This Paper Address?

Biological research generates massive amounts of data, and ontologies are crucial for organizing, annotating, and interpreting it. However, the biological domain is rich with many overlapping ontologies, often developed independently. This leads to incompatibilities, making it difficult to integrate data across different projects or use multiple ontologies together in a modular and coherent way. The Open Biological and Biomedical Ontologies (OBO) Foundry was established to address this by providing a set of overarching principles for developing and sharing ontologies. The problem was that these principles were often subjectively interpreted, making objective evaluation and consistent application difficult.

Methods

The core contribution of this paper is the "operationalization" of the OBO Foundry principles. This involved:

  • Refining Principles: Working with stakeholders, the original OBO principles were re-examined and precisely defined as operational rules.
  • Automated Validation Checks: For each principle, automated checks were developed. These checks are designed to be executable at regular intervals.
  • Dashboard Implementation: A web-based dashboard was created to display the results of these automated checks. This dashboard provides a matrix view, showing each ontology's conformance to each principle. It allows users to drill down for detailed reports on specific ontologies and principles.
  • Metadata Curation: A significant effort was undertaken to curate and update metadata for all ontologies in the OBO registry. This metadata is essential for the automated checks.
  • Collaborative Curation System: A lightweight system using GitHub pull requests was established to ensure the OBO registry records remain up-to-date and collaboratively maintained.

Datasets

The paper evaluates the entire suite of OBO Foundry ontologies. As of May 2021, this includes over 175 active ontologies. The primary "dataset" is the metadata associated with these ontologies, stored in the OBO registry, and the ontologies themselves.

Key Insights and Ideas

  • Operationalizing Principles is Key: Moving from subjective principles to concrete, automated tests is essential for large-scale, federated communities like OBO to achieve consistent quality and interoperability.
  • Community Engagement Drives Progress: The development and refinement of these principles and checks were a collaborative effort, highlighting the importance of stakeholder involvement.
  • Transparency Fosters Improvement: The dashboard's transparent reporting of conformance issues has already led to significant improvements in ontology quality, demonstrating that highlighting problems openly can be a powerful motivator.
  • OBO and FAIR are Synergistic: The goals of the OBO project are highly compatible with the FAIR (Findable, Accessible, Interoperable, Reproducible) data principles, and the lessons learned from OBO can inform FAIR evaluations.

Key Results

  • As of May 2021, four OBO Foundry principles are fully conformed to by all 175 ontologies: FP02 Common Format, FP03 URIs, FP11 Locus of Authority, and FP20 Responsiveness.
  • The principle with the lowest conformance is FP06 Textual Definitions, with only 19 ontologies (approximately 11%) fully passing this check.
  • A preliminary version of the dashboard, announced in early 2020, led to a significant reduction in reported errors by ontology maintainers by July 2020 (p=0.0005).
  • Over 60 updates to OBO registry metadata were made as a result of the initial data collection and confirmation process.
  • The collaborative metadata curation system has seen 3045 commits by 113 developers to the new repository as of May 2021.

Limitations and Future Work

  • Single Ontology Focus: The current framework primarily examines ontologies individually. Future work aims to extend checks to analyze inter-ontology consistency across sets of ontologies.
  • Non-Automatable Principles: Not all OBO principles can be reliably checked automatically. Principles like "scope," "plurality of users," and "co-operation with existing ontologies" still require human review.
  • Continuous Improvement: The authors plan to run the OBO Dashboard on all new ontology membership requests and releases, expecting it to identify pipeline weaknesses and drive continuous improvements in both the automated tests and community understanding of the principles.
  • Community Feedback: The authors explicitly welcome community feedback and criticism on the dashboard and its implementation, seeing it as a crucial part of its ongoing development.

Summary updated

Abstract

Abstract Biological ontologies are used to organize, curate, and interpret the vast quantities of data arising from biological experiments. While this works well when using a single ontology, integrating multiple ontologies can be problematic, as they are developed independently, which can lead to incompatibilities. The Open Biological and Biomedical Ontologies (OBO) Foundry was created to address this by facilitating the development, harmonization, application, and sharing of ontologies, guided by a set of overarching principles. One challenge in reaching these goals was that the OBO principles were not originally encoded in a precise fashion, and interpretation was subjective. Here we show how we have addressed this by formally encoding the OBO principles as operational rules and implementing a suite of automated validation checks and a dashboard for objectively evaluating each ontology’s compliance with each principle. This entailed a substantial effort to curate metadata across all ontologies and to coordinate with individual stakeholders. We have applied these checks across the full OBO suite of ontologies, revealing areas where individual ontologies require changes to conform to our principles. Our work demonstrates how a sizable federated community can be organized and evaluated on objective criteria that help improve overall quality and interoperability, which is vital for the sustenance of the OBO project and towards the overall goals of making data FAIR.
BibTeX
@misc{jackson2021obo,
  author = {Rebecca Jackson and Nicolas Matentzoglu and James A. Overton and Randi Vita and James P. Balhoff and Pier Luigi Buttigieg and Seth Carbon and Mélanie Courtot and Alexander D. Diehl and Damion Dooley and William D. Duncan and Nomi L. Harris and Melissa Haendel and Suzanna Lewis and Darren A. Natale and David Osumi-Sutherland and Alan Ruttenberg and Lynn M. Schriml and Barry Smith and Christian J. Stoeckert and Nicole Vasilevsky and Ramona Walls and Jie Zheng and Chris Mungall and Bjoern Peters},
  title = {{OBO Foundry in 2021: Operationalizing Open Data Principles to Evaluate Ontologies}},
  year = {2021},
  doi = {10.1101/2021.06.01.446587},
  howpublished = {bioRxiv},
}

Loading related content...

Figure