The Cells Out of Sample (COOS) dataset and benchmarks for measuring out-of-sample generalization of image classifiers

Alex X. Lu, Amy X. Lu, Wiebke Schormann, Marzyeh Ghassemi, David W. Andrews, Alan M. Moses

TL;DR

This paper introduces COOS-7, a new dataset and benchmark for evaluating how well image classifiers generalize to data that differs slightly from their training set. They found that even state-of-the-art models struggle significantly when faced with these "out-of-sample" variations, highlighting a critical challenge in deploying AI in real-world scenarios.

What Problem Does This Paper Address?

In machine learning, a classifier's usefulness hinges on its ability to accurately predict outcomes for data it hasn't seen before (out-of-sample data). Traditionally, this is tested by holding out a random subset of the training data. However, this doesn't reflect real-world scenarios where data distributions can subtly shift due to factors like different imaging equipment, varying experimental conditions, or the passage of time. These shifts, known as "covariate shifts," can severely degrade a classifier's performance, with potentially serious consequences in fields like medical diagnostics or drug discovery. Existing datasets often lack the controlled variations needed to systematically measure and address this generalization problem.

Methods

The researchers created a new dataset, COOS-7, consisting of 132,209 microscopy images of mouse cells, each stained with one of seven fluorescent proteins. They then established a benchmark by training and evaluating various classification models on this dataset. The models included:

  • Classic computer vision features: Haralick texture features.
  • Supervised deep CNNs: Specifically, DeepLoc, a 11-layer CNN.
  • Features from pre-trained CNNs: Using VGG16 models trained on ImageNet.
  • Features from self-supervised CNNs: Employing a model called PCI.

For each feature set, they trained three types of classifiers: k-nearest neighbors (kNN), L1 Logistic Regression, and Random Forests. Performance was measured using balanced classification error to account for class imbalances across datasets.

Datasets

  • Training Dataset: 41,456 images from 4 independent plates.
  • Test Datasets:
    • Test1: 10,364 randomly held-out images from the same plates as the training data.
  • Test2: 17,021 images from the same plates but different wells than the training data.
  • Test3: 32,596 images from 2 independent plates, reproduced months later than the training data.
  • Test4: 30,772 images from 1 plate, reproduced on a different day and imaged with a different microscope than the training data.

The COOS-7 dataset is publicly available.

Key Insights and Ideas

  • Covariate Shift is a Major Hurdle: Even with carefully curated, visually distinct classes and controlled experimental variations, significant drops in performance are observed as covariate shifts increase. This suggests that current generalization techniques are insufficient for robust real-world deployment.
  • Representation Quality Matters Most: The study found that the quality of the feature representation was more critical for generalization than the complexity of the classification model itself. For instance, a kNN classifier on good features performed similarly to a complex deep CNN.
  • Non-Uniform Error Patterns: Covariate shifts don't affect all classes equally. Some classes might show a sharp performance decline while others remain relatively unaffected, making overall accuracy metrics potentially misleading.
  • Validation Needs Rigor: Relying on a single out-of-sample dataset for validation is insufficient. The nature of covariate shifts can vary, and a model robust to one type of shift might fail on another.

Key Results

  • All tested classifiers performed well on Test1 and Test2, which had minimal covariate shifts, with errors as low as 1.1%.
  • However, on Test3 and Test4, which introduced greater covariate shifts (time and instrument differences), all classifiers experienced significant performance drops.
  • For example, the logistic regression classifier on VGG16 features showed a balanced classification error of 6.7% on Test2 but jumped to 12.1% on Test4. Similarly, DeepLoc's end-to-end performance went from 1.5% error on Test2 to 7.4% on Test3.
  • Confusion matrices revealed that errors were often concentrated on specific, visually similar classes (e.g., early endosome vs. ER/Golgi), while distinct classes (cytosol, nuclear envelope) were generally classified well.

Limitations and Future Work

  • Controlled Environment: The COOS-7 dataset was designed with visually distinct fluorescent proteins and balanced classes to make it amenable to ML methods. The authors acknowledge that real-world biological imaging might present even greater challenges with more ambiguous classes and complex experimental variability.
  • Future Datasets: The authors plan to release further datasets with more ambiguous classes and a wider range of experimental variability to explore these challenges further.
  • Practical Implications: Improvements in out-of-sample generalization on this dataset have direct implications for biologists needing to automate the labeling of protein localization, which is crucial for understanding cell function and disease.

Summary updated

Abstract

Understanding if classifiers generalize to out-of-sample datasets is a central problem in machine learning. Microscopy images provide a standardized way to measure the generalization capacity of image classifiers, as we can image the same classes of objects under increasingly divergent, but controlled factors of variation. We created a public dataset of 132,209 images of mouse cells, COOS-7 (Cells Out Of Sample 7-Class). COOS-7 provides a classification setting where four test datasets have increasing degrees of covariate shift: some images are random subsets of the training data, while others are from experiments reproduced months later and imaged by different instruments. We benchmarked a range of classification models using different representations, including transferred neural network features, end-to-end classification with a supervised deep CNN, and features from a self-supervised CNN. While most classifiers perform well on test datasets similar to the training dataset, all classifiers failed to generalize their performance to datasets with greater covariate shifts. These baselines highlight the challenges of covariate shifts in image data, and establish metrics for improving the generalization capacity of image classifiers.
BibTeX
@inproceedings{lu2019cells,
  author = {Alex X. Lu and Amy X. Lu and Wiebke Schormann and Marzyeh Ghassemi and David W. Andrews and Alan M. Moses},
  title = {{The Cells Out of Sample (COOS) dataset and benchmarks for measuring out-of-sample generalization of image classifiers}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
}
Figure