Domain Adaptive and Fine-grained Anomaly Detection for Single-cell Sequencing Data and Beyond

Kaichen Xu, Yueyang Ding, Suyang Hou, Weiqiang Zhan, Nisang Chen, Jun Wang, Xiaobo Sun

TL;DR

The paper introduces ACSleuth, a novel generative framework designed for fine-grained anomalous cell detection (FACD) in single-cell sequencing data across multiple samples and domains. By integrating anomaly detection, domain adaptation, and cell subtyping into a unified workflow, it overcomes the pervasive issue of technical batch effects masking true biological variations. Furthermore, the authors provide the first theoretical proof validating the use of reconstruction deviations for anomaly detection under domain shifts.

What Problem Does This Paper Address?

Identifying abnormal cells (such as tumor cells or rare pathogenic variants) from single-cell (SC) sequencing data is crucial for clinical diagnosis and biomedical research. However, doing this across multiple samples introduces a major challenge: Domain Shifts (DS).

Domain shifts are non-biological variations—often called batch effects—caused by differences in sequencing technologies, operators, or experimental conditions. These technical shifts intermingle with true biological variations ("content shifts"), leading to three main types of errors:

  1. False Positives: Normal cells in a target dataset are mislabeled as anomalous due to technical differences from the reference dataset.
  2. Incorrect Separation: Anomalous cells of the same biological type look different across samples due to batch effects, causing them to be misclassified as distinct types.
  3. Unintentional Conflation: Efforts to remove domain shifts accidentally erase the subtle biological differences needed to distinguish between distinct anomaly subtypes.
Figure 1: Three error types in multi-sample and multi-domain FACD analysis due to domain shift and sample-specific AC types. Domain shift represents non-biological variations caused by technical differ- ences among samples, while content shift represents true biological variations.

Existing anomaly detection methods either fail to handle these domain shifts or stop short at simply labeling cells as "anomalous" without categorizing them into pathologically distinct subtypes.

Methods

ACSleuth tackles this via a cohesive, three-phase generative adversarial network (GAN) framework:

Figure 2: The workflow of ACSleuth.
  • Phase I: Anomalous Cell Detection (ACD): A specialized GAN (Module I) featuring a queue-based memory block (to prevent mode collapse) is trained exclusively on normal reference cells. Target cells are passed through this module. Instead of using standard reconstruction error, the authors introduce a novel Maximum Mean Discrepancy (MMD)-based anomaly scorer to translate reconstruction deviations into robust anomaly scores.
  • Phase II: Multi-Sample Domain Adaptation: To strip away technical batch effects without altering biological signals, a second GAN (Module II) identifies "kin" pairs—biologically similar normal cells between the reference and target datasets. It learns a domain shift matrix to map target cells into the reference domain.
  • Phase III: Fine-grained AC Annotation: The domain-adapted cell embeddings are fused with their corresponding reconstruction deviations using a cross-attention fusion block. This enriched representation is then fed into a self-paced deep clustering algorithm to automatically categorize the anomalous cells into fine-grained subtypes.

Datasets

The authors rigorously evaluated ACSleuth across 18 experimental scenarios spanning different domain shift types:

  • scRNA-seq Datasets: 10x Peripheral Blood Mononuclear Cell (PBMC), lung cancer (Cancer), Cutaneous Squamous Cell Carcinoma (CSCC), and Pancreatic (Pan) datasets using various technologies (InDrop, Smart-Seq2, CEL-Seq2).
  • scATAC-seq Datasets: Human Tumor Microenvironment (TME) and Mouse Brain datasets.
  • Beyond Single-Cell Tabular Data: The KDDRev dataset for cyber-intrusion detection, demonstrating the model's generalizability to non-biological tabular data where connection protocols introduce domain shifts.

Key Insights and Ideas

  • Theoretical Grounding for Reconstruction Deviations: The paper provides the first mathematical proof (Theorems 3.1–3.3) showing that reconstruction deviations from generative models can bound domain-associated variations. This justifies why generative reconstruction works for anomaly detection even when domains shift.
  • The Power of Feature Fusion: In Phase III, rather than clustering cells based solely on latent embeddings, ACSleuth fuses embeddings with reconstruction deviations. Because the deviation captures the "unlearnable" anomalous features that the normal-trained generator failed to reconstruct, it provides highly discriminative signals for fine-grained subtyping.
  • Kin-Pair Alignment: Standard domain adaptation methods (like Mutual Nearest Neighbors) can be corrupted when target datasets contain sample-specific anomaly types. By focusing adaptation strictly on "kin" normal pairs, ACSleuth remains resilient against complications caused by unique anomaly types.

Key Results

  • Superior Anomaly Detection: In benchmarks across 17 single-cell experiments, ACSleuth consistently outperformed state-of-the-art anomaly detection methods (such as SLAD, ICL, and CAMLU), ranking first in 13 scenarios and in the top two for 16 scenarios.
  • Unmatched Fine-Grained Annotation: When evaluating the entire workflow (Detection + Subtyping) using the product of F1 score and Normalized Mutual Information ($F1 \times NMI$), ACSleuth achieved an average performance leap of 74.13% over competing baseline methods.
  • Ablation Validation: Ablation studies confirmed the necessity of each component. Omitting the Domain Adaptation phase led to an average decline of 36.49% in $F1 \times NMI$ scores, while replacing the MMD-based scorer with standard scoring functions reduced detection accuracy by up to 22%.

Limitations and Future Work

While ACSleuth demonstrates remarkable versatility and accuracy, a few areas remain open for future exploration:

  • Cluster Number Inference: Although the framework includes a graph Laplacian heuristic to automatically infer the number of anomaly subtypes when unknown, this step can be sensitive to hyperparameter tuning in highly heterogeneous samples.
  • Scalability: The paper does not explicitly detail the computational runtime or memory footprint when scaling to massive, atlas-scale datasets containing millions of cells.
  • Multi-modal Integration: Future work could extend this framework to handle multi-modal single-cell data (e.g., simultaneous scRNA-seq and scATAC-seq from the same cell) to capture even finer pathological nuances.

Summary updated

Abstract

Fined-grained anomalous cell detection from affected tissues is critical for clinical diagnosis and pathological research. Single-cell sequencing data provide unprecedented opportunities for this task. However, current anomaly detection methods struggle to handle domain shifts prevalent in multi-sample and multi-domain single-cell sequencing data, leading to suboptimal performance. Moreover, these methods fall short of distinguishing anomalous cells into pathologically distinct subtypes. In response, we propose ACSleuth, a novel, reconstruction deviation-guided generative framework that integrates the detection, domain adaptation, and fine-grained annotating of anomalous cells into a methodologically cohesive workflow. Notably, we present the first theoretical analysis of using reconstruction deviations output by generative models for anomaly detection in lieu of domain shifts. This analysis informs us to develop a novel and superior maximum mean discrepancy-based anomaly scorer in ACSleuth. Extensive benchmarks over various single-cell data and other types of tabular data demonstrate ACSleuth's superiority over the state-of-the-art methods in identifying and subtyping anomalies in multi-sample and multi-domain contexts. Our code is available at https://github.com/Catchxu/ACsleuth.
BibTeX
@inproceedings{xu2024domain,
  author = {Kaichen Xu and Yueyang Ding and Suyang Hou and Weiqiang Zhan and Nisang Chen and Jun Wang and Xiaobo Sun},
  title = {{Domain Adaptive and Fine-grained Anomaly Detection for Single-cell Sequencing Data and Beyond}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure