Development of a universal imaging “phenome” using Shape, Appearance and Motion (SAM) features and the SAM Observation Tool (SPOT)

Felix Zhou, Adam Norton-Steele, Lewis Marsh, Helen M. Byrne, Heather A. Harrington, Xin LübioRxiv 2025 · DOI

TL;DR

This paper introduces SAM-phenome, a universal set of 2185 image-derived features for characterizing cell behavior, and SPOT, a standardized workflow for analyzing these features in timelapse imaging. This approach offers a comprehensive and interpretable way to study dynamic cellular phenotypes, outperforming deep learning models in certain benchmarks and paving the way for high-throughput biological discovery.

What Problem Does This Paper Address?

Traditional methods for analyzing cell behavior in microscopy, especially timelapse imaging, often lack standardization and comprehensive feature extraction. While single-cell RNA sequencing (scRNA-seq) revolutionized biological discovery by providing a high-dimensional, standardized view of cellular states, image-based analysis has lagged behind. Existing image analysis tools often define custom features for specific experiments, making comparisons difficult. Furthermore, deep learning approaches, while powerful, can be "black boxes," lacking interpretability and requiring extensive training data. This paper addresses the need for a standardized, comprehensive, and interpretable method to extract rich phenotypic information from live-cell timelapse imaging, analogous to the impact of scRNA-seq on genomics.

Methods

The core of this work lies in two key developments:

  • SAM-phenome: A standardized set of 2185 features derived from an object's Shape, Appearance, and Motion. These features are designed to be universal, applicable across different imaging modalities and object types, and are measured at multiple spatial scales (global, regional, and pixelwise). This comprehensive feature set aims to capture a cell's "image-transcriptome."
  • SPOT (SAM Phenotype Observation Tool): A four-stage workflow for analyzing SAM-phenomes:
  • Preprocessing: Videos are converted to a standard format, with steps like 3D-to-2D projection and frame-by-frame registration to remove motion artifacts.
  • Object Detection and Tracking: Objects of interest are segmented and tracked across frames to create consistent tracklets.
  • SAM Feature Computation: The SAM-phenome is computed for each identified object instance at each timepoint.
  • Temporal Analysis: This stage involves discovering phenotype dynamics (e.g., population trajectories, cluster analysis) and interpreting results by identifying modules of covarying SAM features. This stage is heavily inspired by scRNA-seq analysis workflows, using techniques like UMAP for dimensionality reduction and Hidden Markov Models for temporal analysis.

Datasets

The authors validated their SAM-SPOT workflow on several publicly available datasets:

  • Computer Vision Datasets:
    • MPEG-7 Shape Dataset: A benchmark dataset of 1400 shapes across 70 categories, used to test shape feature performance.
  • Normalized Brodatz Texture Database: A dataset of 112 texture images, augmented to over 10,000 patches, used to test appearance feature performance.
  • Combined Brodatz and MPEG-7 Dataset: A synthetic dataset of 10,000 images combining shape and appearance variations, used to test joint SAM-SA features.
  • A2D Dataset: A challenging dataset of over 3500 YouTube videos depicting real-life scenarios with objects performing various movements, used to test SAM features on shape, appearance, and motion in unconstrained environments.
  • Biological Datasets:
    • LIVECell Dataset: A large-scale dataset of over 1.6 million fixed cells from 8 cell lines, used to test SAM features on diverse cell morphologies.
  • U373 and HL60 Cell Tracking Datasets: Two datasets from the 2D single-cell tracking challenge, featuring migrating (U373) and dividing (HL60) cells, used to test SPOT's temporal analysis capabilities.
  • HeLa Cell Histone H2B Dataset: A dataset of timelapse microscopy movies of HeLa cells undergoing mitosis, used to test SAM and CNN-LSTM features on nuclear appearance changes during cell division.

Key Insights and Ideas

  • SAM-phenome as an "Image-Transcriptome": The concept of a universal, high-dimensional feature set for images, analogous to scRNA-seq's transcriptome, is a powerful framing for image-based biological analysis.
  • Universality and Standardization: The SAM-phenome is designed to be universally applicable, reducing the need for experiment-specific feature engineering and enabling cross-dataset comparisons.
  • Interpretability of Features: Unlike many deep learning models, SAM features are derived from interpretable concepts (shape, appearance, motion) and spatial scales, allowing for more direct biological interpretation.
  • SPOT as a Standardized Workflow: The SPOT workflow provides a structured and reproducible pipeline for analyzing timelapse imaging data, democratizing complex video analysis.
  • Outperforming Deep Learning in Specific Tasks: The paper demonstrates that the SAM-phenome can outperform deep learning models, particularly in tasks requiring fine-grained morphological distinctions and interpretability.

Key Results

  • The SAM-phenome comprises 2185 features, categorized by shape, appearance, and motion, and measured at global, regional, and distribution scales.
  • SPOT analysis on computer vision datasets showed that the full SAM-S (kernel ECC) feature set outperformed individual shape feature subgroups and CellProfiler's shape features in clustering accuracy (AMI=0.71, ARI=0.52).
  • SAM-A features were competitive with CellProfiler's appearance features, organizing image patches by textural patterns, whereas CellProfiler emphasized global brightness.
  • The combined SAM-SA features were more discriminative than CellProfiler's SA features in capturing joint shape and appearance variations.
  • On the A2D dataset, the full SAM phenome significantly outperformed SAM-S, SAM-A, and SAM-M features alone in clustering and classification tasks.
  • In biological datasets, SAM-SPOT analysis successfully characterized phenotypic heterogeneity in cell migration (U373) and division (HL60), and nuclear appearance during mitosis (HeLa).
  • Crucially, SAM-phenome outperformed CNN-learned features on large fixed-cell (LIVECell) and video datasets, achieving higher accuracy in cell line prediction (SAM: 0.835 accuracy vs. CNN: 0.694) and mitosis stage classification (SAM: 0.649 accuracy vs. CNN-LSTM: 0.454).

Limitations and Future Work

  • Computational Cost: While SPOT aims for efficiency, processing large video datasets can still be computationally intensive.
  • Segmentation Dependence: The accuracy of SAM features relies on accurate object segmentation and tracking, which can be challenging in complex biological samples.
  • Generalizability of Deep Learning: The paper highlights that while deep learning models can learn from data, they may not always capture the most relevant or interpretable features without careful design and constraints.
  • Future Work: The authors suggest that SPOT is currently implemented for 2D videos but can be extended to 3D organoid analysis from 2D projections. They also envision users augmenting the SAM phenome with additional features from other modalities (e.g., genetics-based) or external software. The development of more sophisticated deep learning architectures that incorporate SAM-aware design principles is also a promising avenue.

Summary updated

Abstract

Abstract Cells are plastic, highly heterogeneous and change over time. High-content timelapse imaging promises to reveal dynamic cell behaviors, enabling more accurate identification of cell state and cell fate prediction for biological hypothesis generation and perturbation screens. To empower live-cell imaging based screen, we report the development of 1) a Shape, Appearance, Motion (SAM) “phenome”; a universal set of 2185 image-derived features that act as a image-“transcriptome” to comprehensively quantify an object’s instantaneous phenotype; 2) the SAM-Phenotype-Observation-Tool (SPOT), for image-“sequencing” analysis of phenomes. We validated the effectiveness of unbiased SAM-SPOT workflow on publicly available computer vision and 2D single cell imaging datasets. Importantly, we demonstrated that SAM-phenome outperformed features generated by deep learning AI models trained on >1 million fixed single cell and >5000 single cell video frames, respectively. SAM-phenome and SPOT delivers high-throughput, object-treatment-agnostic, comprehensive screening readouts of dynamics, promising to advance novel molecular target discovery and new medicine development.
BibTeX
@misc{zhou2025development,
  author = {Felix Zhou and Adam Norton-Steele and Lewis Marsh and Helen M. Byrne and Heather A. Harrington and Xin Lü},
  title = {{Development of a universal imaging “phenome” using Shape, Appearance and Motion (SAM) features and the SAM Observation Tool (SPOT)}},
  year = {2025},
  doi = {10.1101/2025.11.28.691105},
  howpublished = {bioRxiv},
}
Figure