Time-Varying Clusters in Large-Scale Flow Cytometry

Jeremy Hyrkas, Daniel Halperin, Bill Howe

Abstract

Flow cytometry is a cornerstone of cellular analysis, yet its application in environmental monitoring presents unique challenges that traditional algorithms are ill-equipped to handle. This paper introduces the "SeaFlow" challenge: classifying microbial populations in massive, time-varying datasets generated by continuous flow cytometry on oceanographic vessels. We evaluate state-of-the-art medical clustering algorithms, demonstrating their failure to manage environmental "drift" and the massive scale of oceanic data. This work frames the need for robust, scalable, and time-aware machine learning models as a significant open problem for the AI community.

Introduction: The SeaFlow Challenge

Flow cytometry measures the physical and chemical characteristics of cells as they pass through a laser. While traditionally used in controlled medical settings, the SeaFlow instrument brings this technology to the open ocean. Mounted on research vessels, SeaFlow operates continuously for weeks, analyzing millions of phytoplankton particles as the ship traverses different water masses.

This transition from the lab to the field introduces three major hurdles:

  1. Time-Variance: Environmental shifts in temperature and nutrients cause microbes to grow, shrink, or change pigment concentrations, leading to "clusters" that shift and evolve over time.
  2. Massive Scale: A single cruise generates hundreds of gigabytes of data, far exceeding the capacity of single-threaded medical tools.
  3. High Noise: Unlike lab settings, ocean samples contain diverse organic matter, creating significant measurement uncertainty.

Methods and Problem Characterization

The study primarily serves as a problem characterization, evaluating how existing unsupervised clustering algorithms—specifically flowMeans and flowPeaks—perform on environmental data.

The authors identify two common but flawed approaches:

  • Global Clustering: Treating multi-week datasets as a single unit fails because populations vary too significantly across different water masses.
  • Windowed Clustering: Treating each 3-minute window independently ignores the vital temporal continuity of biological populations.

The proposed direction involves structured learning to map cluster layouts in 6D space directly from sequences of windows, potentially leveraging semi-supervised learning to utilize small amounts of high-quality labeled data.

Datasets

The research utilizes 17 SeaFlow datasets provided by the University of Washington. Each dataset contains millions of particles measured in a 6-dimensional optical space, including forward scatter and various fluorescence wavelengths. Three specific datasets (Thompson cruises 0, 1, and 11) serve as "gold standards," having been highly curated by scientists to provide ground truth for classification performance. These datasets are now available to the research community to foster innovation.

Results: The Medical Gap

The core finding is a significant performance drop when applying medical-grade algorithms to environmental data. In medical benchmarks (such as FlowCAP), algorithms like flowMeans and flowPeaks typically achieve F-measures above 0.9. However, on SeaFlow data, performance is both lower and highly inconsistent.

Cumulative Distribution Function (CDF) of F-measures for flowMeans and flowPeaks across SeaFlow datasets.

As shown in the figure above, F-measures frequently dip into the 0.4 to 0.7 range. While flowMeans generally outperforms flowPeaks, neither provides the reliability required for automated scientific analysis. Furthermore, these algorithms are not designed for the data-parallel, distributed environments necessary to process 100+ GB datasets efficiently.

Discussion and Future Work

The "biological drift" observed—where a species' optical signature changes based on time of day or geographic location—highlights that the "shape" of a species in 6D space is not a constant. This makes manual "gating" (classification by inspection) unsustainable.

Future work must focus on:

  • Tracking Algorithms: Developing methods that can track clusters as they evolve temporally.
  • Scalable Architectures: Adapting models like Gaussian Mixture Models (GMMs) for distributed, massive-scale environments.
  • Multi-Instrument Integration: Enabling simultaneous processing from multiple vessels to ensure robust calibration across global deployments.

By addressing these gaps, the AI community can provide the tools necessary for real-time, large-scale monitoring of the world's oceans.

Summary updated

Abstract

Flow cytometers measure the optical properties of particles to classify microbes. Recent innovations have allowed oceanographers to collect flow cytometry data continuously during research cruises, leading to an explosion of data and new challenges for the classification task. The massive scale, time-varying underlying populations, and noisy measurements motivate the development of new classification methods. We describe the problem, the data, and some preliminary results demonstrating the difficulty with conventional methods.
BibTeX
@inproceedings{hyrkas2015time,
  author = {Jeremy Hyrkas and Daniel Halperin and Bill Howe},
  title = {{Time-Varying Clusters in Large-Scale Flow Cytometry}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2015},
  doi = {10.1609/aaai.v29i2.19067},
}
Figure