TL;DR
This paper tackles the challenge of integrating diverse biological ontologies by operationalizing the Open Biological and Biomedical Ontologies (OBO) Foundry's guiding principles into automated checks. They've developed a dashboard that objectively evaluates ontologies against these principles, aiming to improve their quality, interoperability, and alignment with FAIR data standards.
What Problem Does This Paper Address?
Biological research generates massive amounts of data, and ontologies are crucial for organizing, annotating, and interpreting it. However, the biological domain is rich with many overlapping ontologies, often developed independently. This leads to incompatibilities, making it difficult to integrate data across different projects or use multiple ontologies together in a modular and coherent way. The Open Biological and Biomedical Ontologies (OBO) Foundry was established to address this by providing a set of overarching principles for developing and sharing ontologies. The problem was that these principles were often subjectively interpreted, making objective evaluation and consistent application difficult.
Methods
The core contribution of this paper is the "operationalization" of the OBO Foundry principles. This involved:
- Refining Principles: Working with stakeholders, the original OBO principles were re-examined and precisely defined as operational rules.
- Automated Validation Checks: For each principle, automated checks were developed. These checks are designed to be executable at regular intervals.
- Dashboard Implementation: A web-based dashboard was created to display the results of these automated checks. This dashboard provides a matrix view, showing each ontology's conformance to each principle. It allows users to drill down for detailed reports on specific ontologies and principles.
- Metadata Curation: A significant effort was undertaken to curate and update metadata for all ontologies in the OBO registry. This metadata is essential for the automated checks.
- Collaborative Curation System: A lightweight system using GitHub pull requests was established to ensure the OBO registry records remain up-to-date and collaboratively maintained.
Datasets
The paper evaluates the entire suite of OBO Foundry ontologies. As of May 2021, this includes over 175 active ontologies. The primary "dataset" is the metadata associated with these ontologies, stored in the OBO registry, and the ontologies themselves.
Key Insights and Ideas
- Operationalizing Principles is Key: Moving from subjective principles to concrete, automated tests is essential for large-scale, federated communities like OBO to achieve consistent quality and interoperability.
- Community Engagement Drives Progress: The development and refinement of these principles and checks were a collaborative effort, highlighting the importance of stakeholder involvement.
- Transparency Fosters Improvement: The dashboard's transparent reporting of conformance issues has already led to significant improvements in ontology quality, demonstrating that highlighting problems openly can be a powerful motivator.
- OBO and FAIR are Synergistic: The goals of the OBO project are highly compatible with the FAIR (Findable, Accessible, Interoperable, Reproducible) data principles, and the lessons learned from OBO can inform FAIR evaluations.
Key Results
- As of May 2021, four OBO Foundry principles are fully conformed to by all 175 ontologies: FP02 Common Format, FP03 URIs, FP11 Locus of Authority, and FP20 Responsiveness.
- The principle with the lowest conformance is FP06 Textual Definitions, with only 19 ontologies (approximately 11%) fully passing this check.
- A preliminary version of the dashboard, announced in early 2020, led to a significant reduction in reported errors by ontology maintainers by July 2020 (p=0.0005).
- Over 60 updates to OBO registry metadata were made as a result of the initial data collection and confirmation process.
- The collaborative metadata curation system has seen 3045 commits by 113 developers to the new repository as of May 2021.
Limitations and Future Work
- Single Ontology Focus: The current framework primarily examines ontologies individually. Future work aims to extend checks to analyze inter-ontology consistency across sets of ontologies.
- Non-Automatable Principles: Not all OBO principles can be reliably checked automatically. Principles like "scope," "plurality of users," and "co-operation with existing ontologies" still require human review.
- Continuous Improvement: The authors plan to run the OBO Dashboard on all new ontology membership requests and releases, expecting it to identify pipeline weaknesses and drive continuous improvements in both the automated tests and community understanding of the principles.
- Community Feedback: The authors explicitly welcome community feedback and criticism on the dashboard and its implementation, seeing it as a crucial part of its ongoing development.
Summary updated