The Cells Out of Sample (COOS) dataset and benchmarks for measuring out-of-sample generalization of image classifiers

Alex X. Lu, Amy X. Lu, Wiebke Schormann, Marzyeh Ghassemi, David W. Andrews, Alan M. Moses

TL;DR

The COOS-7 dataset provides a new, rigorous benchmark for testing how well image classifiers handle "covariate shift"—the subtle changes in data caused by different equipment or timing. The researchers found that while modern AI models excel on standard "random-split" tests, their performance collapses when faced with realistic variations, such as images taken on a different microscope months later.

What Problem Does This Paper Address?

In machine learning, we usually evaluate models by holding out a random subset of data for testing. If the model performs well on that subset, we assume it has "learned" the task. However, this is a bit of a cheat: the test data comes from the exact same distribution as the training data.

In the real world—especially in biology and medicine—data is messy. A model trained on images from Microscope A on Monday might fail on Microscope B on Friday due to subtle changes in lighting, temperature, or hardware calibration. This is known as covariate shift. This paper addresses the lack of standardized datasets designed to measure exactly how much these shifts degrade model performance.

Datasets

The authors introduced COOS-7 (Cells Out Of Sample 7-Class), a dataset of 132,209 images of mouse cells. The goal is to classify which of the seven organelles (like the nucleus or Golgi apparatus) is being highlighted.

Examples from the COOS-7 dataset showing visual variations across different test splits.

What makes COOS-7 unique is its four-tiered testing structure designed to simulate real-world data shifts:

  • Test 1 (Random): Randomly held-out images from the same plates as the training set.
  • Test 2 (Well-shift): Images from different wells on the same plates.
  • Test 3 (Time-shift): Images from experiments reproduced months later.
  • Test 4 (Equipment-shift): Images reproduced on a different day and using a different microscope.

Methods

The authors benchmarked four distinct approaches to feature extraction to see which (if any) could withstand covariate shifts:

  1. Classic Computer Vision: Haralick texture features (a traditional way to describe image patterns).
  2. Supervised Deep Learning: "DeepLoc," a custom 11-layer CNN trained specifically on biological images.
  3. Transfer Learning: Features extracted from a VGG16 model pre-trained on ImageNet (natural images like dogs and cars).
  4. Self-Supervised Learning: Features from a "PCI" model, which learns to represent images without needing manual labels.

On top of these features, they tested standard classifiers like k-Nearest Neighbors (kNN), Logistic Regression, and Random Forests.

Key Insights and Ideas

  • The "Generalization Gap" is Real: Models that look nearly perfect (~1% error) on standard tests (Test 1) see their error rates jump by 4x to 10x when moved to Test 4.
  • Complexity Doesn't Equal Robustness: Simply using a bigger or more complex model didn't solve the problem. The authors found that performance was bounded by the quality of the representation (how the model "sees" the cell), not the classifier sitting on top of it.
  • Errors are Non-Uniform: Covariate shift doesn't just make a model "slightly worse" across the board. Instead, it often causes the model to catastrophically confuse specific, visually similar classes (like mistaking an "early endosome" for the "Golgi") while remaining perfectly fine at identifying others (like the "nuclear envelope").
  • Biological "Toy" Problems are Still Hard: The authors intentionally chose visually distinct cell parts to make the task "easy," yet even then, no current state-of-the-art technique could fully compensate for the shift in instruments.

Key Results

The quantitative drop-off across the different test sets is stark, demonstrating the severe impact of covariate shift.

Classification error rates across different feature extraction methods and test sets, highlighting the generalization gap.

Looking at the DeepLoc model (the best performer):

  • Training/Test 1 Error: ~1.2%
  • Test 3 (Time Shift) Error: 7.4%
  • Test 4 (Microscope Shift) Error: 5.4%

Interestingly, Test 3 was sometimes harder than Test 4, suggesting that time-based biological variation can be as significant as hardware changes. For the Texture-based models, the failure was even more dramatic, with error rates climbing from ~6% on easy tests to over 25% on the hardest test.

Limitations and Future Work

The authors acknowledge that COOS-7 is a relatively "clean" dataset because the seven classes are visually distinct to the human eye. In real-world drug discovery or diagnostics, the differences between classes are often much more subtle.

Future directions include: * Expanding the dataset to include more ambiguous and visually similar cell classes. * Developing new "domain adaptation" techniques that can specifically account for the technical biases of different microscopes. * Using COOS-7 as a standard benchmark to evaluate whether new self-supervised learning techniques actually improve real-world robustness or just "overfit" more sophisticatedly.

Summary updated

Abstract

Understanding if classifiers generalize to out-of-sample datasets is a central problem in machine learning. Microscopy images provide a standardized way to measure the generalization capacity of image classifiers, as we can image the same classes of objects under increasingly divergent, but controlled factors of variation. We created a public dataset of 132,209 images of mouse cells, COOS-7 (Cells Out Of Sample 7-Class). COOS-7 provides a classification setting where four test datasets have increasing degrees of covariate shift: some images are random subsets of the training data, while others are from experiments reproduced months later and imaged by different instruments. We benchmarked a range of classification models using different representations, including transferred neural network features, end-to-end classification with a supervised deep CNN, and features from a self-supervised CNN. While most classifiers perform well on test datasets similar to the training dataset, all classifiers failed to generalize their performance to datasets with greater covariate shifts. These baselines highlight the challenges of covariate shifts in image data, and establish metrics for improving the generalization capacity of image classifiers.
BibTeX
@inproceedings{lu2019cells,
  author = {Alex X. Lu and Amy X. Lu and Wiebke Schormann and Marzyeh Ghassemi and David W. Andrews and Alan M. Moses},
  title = {{The Cells Out of Sample (COOS) dataset and benchmarks for measuring out-of-sample generalization of image classifiers}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
}
Figure