BIAFLOWS: A collaborative framework to reproducibly deploy and benchmark bioimage analysis workflows

Ulysse Rubens, Romain Mormont, Lassi Paavolainen, Volker Bäcker, Gino Michiels, Benjamin Pavie, Leandro A. Scholz, Martin Maška, Devrim Ünay, Graeme Ball, Renaud Hoyoux, Rémy Vandaele, Ofra Golani, Anatole Chessel, Stefan G. Stanciu, Nataša Sladoje, Perrine Paul‐Gilloteaux, Raphaël Marée, Sébastien Tosi

TL;DR

BIAFLOWS is a new open-source web platform designed to address the reproducibility crisis in bioimage analysis. It allows researchers to easily deploy, share, and benchmark image analysis workflows, ensuring that scientific findings can be reliably reproduced and that the best methods can be identified.

What Problem Does This Paper Address?

The increasing complexity of bioimage analysis methods, often involving combinations of image processing and deep learning, makes it difficult to describe and reproduce them using traditional written protocols. Software implementations are frequently shared as poorly documented, non-reusable code, lacking consistent test images and clear baselines for valid results. This hinders reproducibility, makes it hard to adapt methods to new datasets, and leads to duplicated efforts in the research community. While challenges exist, they are often isolated, rely on ad-hoc formats, and predominantly focus on medical imaging, neglecting the needs of biological research. Consequently, end-users struggle to validate their analyses, and developers find it challenging to systematically benchmark and share their work in a reproducible manner.

Methods

BIAFLOWS is a community-driven, open-source web platform built upon Cytomine, a platform for collaborative annotation. Its core innovation lies in its ability to:

  • Batch import and organize annotated image datasets into "BIA problems."
  • Encapsulate versioned bioimage analysis workflows regardless of their original platform (e.g., ImageJ, CellProfiler, Python scripts). This is achieved by packaging workflows and their execution environments into Docker images, ensuring consistent and reproducible runs.
  • Remotely visualize images and workflow results through a web interface.
  • Automatically compute benchmark metrics to assess workflow performance against ground truth data.

The platform leverages DockerHub for storing and versioning workflow images, and can be deployed on various computational resources, including high-performance computing clusters, by converting Docker images to formats like Singularity. It also supports local deployments for private data management and processing.

Datasets

The paper highlights the use of several datasets, primarily sourced from existing challenges and public repositories, to populate the BIAFLOWS online instance. These include:

  • 15 image datasets illustrating 9 major classes of bioimage analysis problems.
  • Datasets from challenges such as DIADEM, Cell Tracking Challenge, Particle Tracking Challenge, and Kaggle Data Science Bowl 2018.
  • Data generated from synthetic data generators (e.g., CytoPacq, TREES toolbox, Vascusynth, SIMCEP).
  • A synthetic dataset of 10 images specifically generated for the nuclei segmentation benchmarking case study.
  • A subset of 65 images from the Science Bowl 2018 nuclei segmentation challenge.

The paper emphasizes that BIAFLOWS is designed to integrate with existing datasets and encourages the community to contribute new ones.

Key Insights and Ideas

  • Reproducibility as a core pillar: BIAFLOWS directly tackles the reproducibility crisis by standardizing workflow deployment and execution through containerization (Docker images).
  • Bridging the gap between developers and users: The platform serves as a common ground for algorithm developers to benchmark their tools and for biologists to access and utilize state-of-the-art analysis methods.
  • Democratizing bioimage analysis: By providing a user-friendly web interface, BIAFLOWS makes complex analysis workflows accessible to a wider scientific audience, reducing the need for specialized computational expertise.
  • Benchmarking as a driver for improvement: The automated benchmarking capabilities allow for fair comparison of different workflows, fostering a competitive environment that drives the development of higher-quality and more robust bioimage analysis techniques.
  • Community-driven development: The platform is designed to be collaborative, encouraging contributions of both data and workflows from the scientific community.

Key Results

The paper presents a case study on benchmarking 7 nuclei segmentation workflows, encompassing classical and deep learning techniques. While specific quantitative results (e.g., accuracy scores) are not detailed in the provided text, the study identified several key trends:

  • Optimal image pre-processing is crucial for achieving high object segmentation accuracy, even when competing with deep learning methods.
  • Deep learning workflows generally outperform classical methods on heterogeneous datasets.
  • Object de-clumping is a critical post-processing step for accurate object segmentation.
  • The study advocates for using a set of benchmarking metrics rather than a single metric (like DICE coefficient) for a more meaningful comparison.
  • Visual inspection of workflow results is often necessary to fully understand the limitations highlighted by metric scores.

All results presented in the case study are reproducible online through the BIAFLOWS platform.

Limitations and Future Work

  • Limited scope of current datasets: While the platform is populated with a growing collection of datasets, the paper acknowledges that the majority of challenge datasets still come from medical imaging, and there's a need to expand biological imaging datasets.
  • Encouraging community contributions: The paper calls for contributions of annotated images and workflows to further enrich the platform's content.
  • Expanding problem classes: Future work includes supporting new problem classes, such as the detection of blinking events in super-resolution microscopy and landmark detection for image registration.
  • Broader applicability: The authors suggest that BIAFLOWS can be adopted in other fields where image analysis is critical, such as material science.
  • Federated platform vision: The ultimate goal is to establish BIAFLOWS as a central, federated platform for sharing scientific images, workflows, and associated knowledge, moving beyond individualistic practices.

Summary updated

Abstract

Abstract Automated image analysis has become key to extract quantitative information from scientific microscopy bioimages, but the methods involved are now often so refined that they can no longer be unambiguously described using written protocols. We introduce BIAFLOWS, a software tool with web services and a user interface specifically designed to document, interface, reproducibly deploy, and benchmark image analysis workflows. BIAFLOWS allows image analysis workflows to be compared fairly and shared in a reproducible manner, safeguarding research results and promoting the highest quality standards in bioimage analysis. A curated instance of BIAFLOWS is available online; it is currently populated with 34 workflows that can be triggered to process image datasets illustrating 15 common bioimage analysis problems organized in 9 major classes. As a complete case study, the open benchmarking of 7 nuclei segmentation workflows, including classical and deep learning techniques, was performed on this online instance. All the results presented can be reproduced online.
BibTeX
@article{rubens2020biaflows,
  author = {Ulysse Rubens and Romain Mormont and Lassi Paavolainen and Volker Bäcker and Gino Michiels and Benjamin Pavie and Leandro A. Scholz and Martin Maška and Devrim Ünay and Graeme Ball and Renaud Hoyoux and Rémy Vandaele and Ofra Golani and Anatole Chessel and Stefan G. Stanciu and Nataša Sladoje and Perrine Paul‐Gilloteaux and Raphaël Marée and Sébastien Tosi},
  title = {{BIAFLOWS: A collaborative framework to reproducibly deploy and benchmark bioimage analysis workflows}},
  journal = {Patterns},
  year = {2020},
  doi = {10.1016/j.patter.2020.100040},
}
Figure