Datathons and Software to Promote Reproducible Research

Leo Anthony Celi, Sharukh Lokhandwala, Robert A. Montgomery, Christopher Moses, Tristan Naumann, Tom Pollard, Daniel Spitz, Robert StretchJournal of Medical Internet Research 2016 · DOI

TL;DR

This paper introduces "Chatto," an open-source software suite designed to streamline interdisciplinary collaboration in medical research and significantly improve research reproducibility. By integrating tools for communication, code management, and data analysis, Chatto aims to overcome common barriers in collaborative data mining projects.

What Problem Does This Paper Address?

The paper tackles a critical issue in scientific research: the lack of reproducibility and the challenges of interdisciplinary collaboration, particularly in data-intensive fields like medicine. Despite the growing availability of large datasets (like electronic health records), clinicians often lack the data science expertise to effectively utilize them. Furthermore, the scientific community faces significant hurdles in reproducing research findings due to issues like time-intensive setup, inadequate documentation, code rot, and a lack of incentives for replication. This paper argues that traditional methods are insufficient to address these gaps and proposes a novel approach to foster collaboration and enhance reproducibility.

Methods

The core of the paper's contribution is the introduction of Chatto, an open-source software suite developed for the Critical Care Datathon. Chatto is designed to facilitate teamwork and reproducibility through several key components:

  • Project Website: A central hub for registering and listing datathon projects.
  • Slack Integration: A group chat service for real-time communication and file sharing among team members.
  • GitHub Integration: For source code control, enabling public code repositories to be automatically created for each team.
  • Jupyter Notebooks: Browser-based interactive coding environments for data analysis and documentation, preconfigured to connect to cloud-hosted databases.
  • Data Transformation Library: An open-source library for connecting to data sources and transforming data, with the ability to create and share new transformations.

The datathon itself involved multidisciplinary teams (clinicians and data scientists) working together over two days to formulate research questions, write code, analyze data, and refine their queries in real-time. Projects were then presented to a panel of judges.

Datasets

The primary dataset used in this study was the Medical Information Mart for Intensive Care (MIMIC-III) database. MIMIC-III is an open-access database containing data on over 60,000 ICU admissions from 2002 through 2012. The paper notes that Chatto's connectors can be adapted for other large research databases.

Key Insights and Ideas

  • Datathons as a Model for Collaboration: The paper reinforces the value of "datathons" as a successful model for bringing together diverse expertise (clinicians, statisticians, data scientists) to tackle complex clinical questions.
  • Chatto's Dual Purpose: Chatto effectively serves two crucial roles: facilitating interdisciplinary teamwork during the research process and promoting research reproducibility post-publication by providing a runnable environment for independent investigators.
  • Reducing Setup Time: A significant benefit of Chatto was its ability to drastically reduce the time required to configure research environments, from hours or days down to minutes.
  • Bridging the Expertise Gap: By providing integrated tools and pre-configured environments, Chatto helps bridge the gap between clinicians' domain knowledge and data scientists' technical skills.
  • Reproducibility as a Responsibility: The paper strongly advocates for a shift in scientific culture where the responsibility for ensuring analytical code is accessible, interpretable, and runnable falls increasingly on the original investigators. Studies failing to meet these standards should be viewed with skepticism.

Key Results

  • Time Savings: Chatto enabled teams to reduce the time spent configuring their research environments to just a few minutes, a process that would typically take hours to days.
  • Teamwork Facilitation: The integration of Slack and GitHub facilitated seamless communication, collaboration, and version control for team discussions, datasets, and analytical code.
  • Post-Datathon Utility: Chatto continued to be a valuable research tool for participants even after the datathon concluded.
  • Publication Success: Of the 8 teams formed, 6 subsequently submitted and had abstracts accepted to the American Thoracic Society international conference, highlighting the productivity of the datathon and the Chatto platform.

Limitations and Future Work

The authors acknowledge several limitations and suggest future directions:

  • Data Sharing within Teams: Teams sometimes divided labor based on expertise, leading to the need to export intermediate results (e.g., to CSV files) for sharing, which could be cumbersome.
  • Simultaneous Editing: The current Jupyter notebook implementation does not allow for simultaneous editing by multiple team members, which could improve collaborative workflow.
  • Jupyter Notebook Source Control: Jupyter notebooks were not automatically source-controlled. Future integration with GitHub for hosted notebooks would simplify code sharing.
  • Centralized Documentation: The lack of a single, centralized location for documentation for MIMIC-III and integrated services presented a challenge for some participants. Future datathons should address this with simplified, centralized documentation.

The authors express hope that Chatto will solve many challenges in collaborative data mining and improve research reproducibility, encouraging a more open and verifiable scientific process.

Summary updated

Abstract

This suite of tools fulfills two purposes: (1) facilitation of interdisciplinary teamwork through archiving and version control of datasets, analytical code, and team discussions, and (2) advancement of research reproducibility by functioning postpublication as an online environment in which independent investigators can rerun or modify analyses with relative ease. With the introduction of Chatto, we hope to solve a variety of challenges presented by collaborative data mining projects while improving research reproducibility.
BibTeX
@article{celi2016datathons,
  author = {Leo Anthony Celi and Sharukh Lokhandwala and Robert A. Montgomery and Christopher Moses and Tristan Naumann and Tom Pollard and Daniel Spitz and Robert Stretch},
  title = {{Datathons and Software to Promote Reproducible Research}},
  journal = {Journal of Medical Internet Research},
  year = {2016},
  doi = {10.2196/jmir.6365},
}
Figure