CyclOps: Cyclical development towards operationalizing ML models for health

Amrit Krishnan, Vallijah Subasri, Kaden McKeen, Ali Kore, Franklin Ogidi, Mahshid Alinoori, Nadim Lalani, Azra Dhalla, Amol A. Verma, Fahad Razak, Deval Pandya, Elham DolatabadimedRxiv 2022 · DOI

TL;DR

This paper introduces CyclOps, an open-source Python framework designed to streamline the development, deployment, and monitoring of machine learning (ML) models in healthcare. By integrating MLOps best practices and providing researcher-friendly APIs, CyclOps aims to accelerate the adoption of ML in clinical settings, addressing the current fragmentation and maintenance issues in existing tools.

What Problem Does This Paper Address?

Developing and deploying ML models in healthcare is a complex and often fragmented process. Researchers and engineers face challenges with poorly maintained tools, a lack of standardized workflows, and difficulties in rigorously evaluating and monitoring models in real-world clinical environments. This leads to slow adoption of potentially life-saving AI technologies. The paper highlights that existing tools are often designed for specific research use cases, lack software engineering rigor, and struggle to scale with large healthcare datasets. CyclOps aims to bridge this gap by providing a unified, extensible, and robust framework that supports the entire ML lifecycle, from data ingestion to model monitoring.

Methods

CyclOps is built around a cyclical development philosophy, emphasizing iterative improvement of data pipelines, model development, and monitoring. The framework is implemented in Python and offers a suite of APIs:

  • Query API: Facilitates data extraction from various health databases (including OMOP and MIMIC-IV) using SQLAlchemy for robust querying and returning data as Pandas DataFrames.
  • Process API: Handles data transformation and feature engineering, offering modules for cleaning, aggregation, normalization, imputation, slicing, dataset splitting, and vectorization. It supports both static (TabularFeatures) and temporal (TemporalFeatures) data.
  • Monitor API: Enables rigorous evaluation and monitoring of deployed ML models. It includes tools for detecting data drift and distribution shifts, crucial for maintaining model performance over time. This API supports both synthetic and clinical shift applications, experiment selection (sensitivity tests, rolling windows), and drift detection using reducers and testers.

The framework emphasizes software engineering best practices like modularity, testability, and continuous integration.

Datasets

The paper demonstrates CyclOps' usability with the following datasets:

  • GEMINI: A multi-center database from general internal medicine units in the Greater Toronto Area, used for in-hospital mortality risk prediction and mortality decompensation prediction.
  • MIMIC-IV: A widely used ICU patient database, also used for mortality decompensation prediction.
  • NIH Chest X-Ray Dataset: A large dataset of frontal-view X-ray images used to demonstrate drift detection in medical imaging.

The paper does not introduce new benchmarks but showcases how CyclOps can be applied to existing, well-established datasets.

Key Insights and Ideas

  • Cyclical Development is Key: The core idea is that ML model development in healthcare isn't a linear process. CyclOps embraces an iterative, cyclical approach, acknowledging that data, models, and monitoring systems need continuous refinement.
  • MLOps for Healthcare: The paper strongly advocates for applying MLOps principles to healthcare ML, recognizing that robust engineering practices are as critical as the ML algorithms themselves for successful clinical deployment.
  • Researcher-Friendly APIs: A significant focus is on making the framework accessible to researchers, providing intuitive Python APIs that abstract away much of the underlying complexity of data handling and pipeline construction.
  • Extensibility for Multimodal Data: CyclOps is designed to handle various data types, including tabular, temporal, and imaging data, making it versatile for diverse clinical applications.
  • Drift Detection is Non-Negotiable: The inclusion of a comprehensive Monitor API underscores the critical importance of detecting and managing data drift in healthcare, where model performance degradation can have serious consequences.

Key Results

The paper presents proof-of-concept experiments demonstrating CyclOps' capabilities:

  • In-hospital mortality risk prediction (GEMINI): A Gradient Boosted Tree (GBT) model showed good performance on in-distribution data, but its performance deteriorated when evaluated on out-of-distribution data (e.g., academic hospitals vs. community hospitals), highlighting the need for drift detection.
  • Mortality decompensation prediction (GEMINI & MIMIC-IV): An LSTM model was developed and evaluated, with drift detection experiments showing a clear shift detected around the declaration of the COVID-19 pandemic, demonstrating the framework's ability to identify real-world data shifts.
  • NIH Chest X-Ray Dataset: Experiments using synthetic shifts (Gaussian noise, categorical metadata changes) showed that the drift detection pipeline could effectively identify and quantify these shifts, even with complex imaging data.

Limitations and Future Work

  • Early Stage Framework: CyclOps is presented as a relatively new framework, and while it demonstrates early usability, further development and community adoption are expected.
  • Generalizability: While designed for broad applicability, the paper acknowledges that specific use cases might require custom extensions or wrapper code for arbitrary databases.
  • Future Work: The authors express a desire to support and maintain the software as it grows, refactoring towards improved usability and functionality. They also plan to extend support for other popular research datasets. The framework's design inherently supports ongoing research into new ML techniques and MLOps strategies for healthcare.

Summary updated

Abstract

A bstract Open source software that enable research and development of machine learning (ML) models for clinical use cases are fragmented, poorly maintained and fall short in functionality. CyclOps is a software framework designed to address this gap and help accelerate the development of ML models for health. In this paper, we describe the architecture, APIs and implementation details of CyclOps, while providing benchmarks on example clinical use cases. We emphasize that CyclOps is developed to be researcher friendly, while providing APIs for building end-to-end pipelines for model development as well as deployment. We adopt software engineering and ML operations (MLOps) best practices, while providing support for handling large volumes of health data. The design of the framework is centered around the notion of iterative and cyclical development of the overall ML system, which consists of data, model development and monitoring pipelines. The core CyclOps package can be installed through the Python Package Index (PyPI) and the source code is available at https://github.com/VectorInstitute/cyclops .
BibTeX
@misc{krishnan2022cyclops,
  author = {Amrit Krishnan and Vallijah Subasri and Kaden McKeen and Ali Kore and Franklin Ogidi and Mahshid Alinoori and Nadim Lalani and Azra Dhalla and Amol A. Verma and Fahad Razak and Deval Pandya and Elham Dolatabadi},
  title = {{CyclOps: Cyclical development towards operationalizing ML models for health}},
  year = {2022},
  doi = {10.1101/2022.12.02.22283021},
  howpublished = {medRxiv},
}
Figure