Domain Adaptive and Fine-grained Anomaly Detection for Single-cell Sequencing Data and Beyond

Kaichen Xu, Yueyang Ding, Suyang Hou, Weiqiang Zhan, Nisang Chen, Jun Wang, Xiaobo Sun

Introduction

In bioinformatics, single-cell sequencing provides high-resolution insights into cellular biology. However, identifying anomalous cells—such as rare disease subtypes—is complicated by "domain shifts" (DS). These technical variations, stemming from different lab conditions or sequencing technologies, often mask true biological signals. This paper introduces ACSleuth, a unified framework presented at IJCAI-24, designed to detect, adapt to, and categorize anomalous cells (ACs) with high precision.

The Challenge of Fine-grained Anomaly Detection

The authors focus on Fine-grained Anomalous Cell Detection (FACD). The goal is to identify cells in a target sample that deviate from a "normal" reference and then categorize these anomalies into distinct biological subtypes.

Conceptual challenges in anomaly detection and annotation under domain shift.

As illustrated in Figure 1, domain shifts create three primary hurdles:

  1. Case i: False Positives: Technical noise causes normal cells in a target dataset to appear anomalous relative to the reference.
  2. Case ii: Subtype Separation: Domain shifts can split the same anomaly type into separate clusters.
  3. Case iii: Conflation: Distinct anomaly types may be unintentionally grouped together due to content shifts.

The ACSleuth Framework

ACSleuth addresses these issues through a three-phase generative workflow. It leverages a specialized Generative Adversarial Network (GAN) and a novel theoretical approach to "reconstruction deviations."

The three-phase architecture of the ACSleuth framework.

Phase I: Anomaly Detection (ACD)

The framework first trains a GAN (Module I) with a memory block to learn the distribution of normal reference cells. A novel MMD-based anomaly scorer evaluates "reconstruction deviations"—the difference between a real cell and its GAN-reconstructed "normal" counterpart. This deviation serves as a robust proxy for identifying anomalies even amidst noise.

Phase II: Multi-sample Domain Adaptation

To handle multiple target datasets, ACSleuth employs a second GAN module to identify "kin" pairs (similar normal cells) across samples. By learning a specific Domain Shift (DS) matrix, the model can "subtract" technical noise without erasing critical biological features.

Phase III: Fine-grained Annotation

Finally, latent features and reconstruction deviations are fused via a cross-attention block. This enriched representation is processed by a self-paced deep clustering algorithm to identify specific AC subtypes (e.g., different tumor stages).

Experimental Evaluation

ACSleuth was evaluated across 18 experimental scenarios, including scRNA-seq (PBMC, lung cancer, skin cancer), scATAC-seq (mouse brain), and the KDDRev cyber-intrusion dataset.

Key findings include:

  • Superior Detection: ACSleuth ranked first in AUC in 13 out of 17 single-cell experiments, outperforming state-of-the-art methods like scGAD and MARS.
  • Enhanced Subtyping: In combined detection and subtyping metrics (F1×NMI), ACSleuth achieved an average improvement of 74.13% over baselines.
  • Robustness: The model proved resilient when target datasets contained unique anomaly types not present in other samples.

Conclusion

ACSleuth represents a significant advancement in single-cell analysis by treating domain shift as a "style" that can be separated from biological "content." By utilizing reconstruction deviations as a primary signal, it provides a more robust mechanism for fine-grained anomaly detection. Future work may extend this framework to multi-modal data or reference-free detection scenarios.

Code Availability: https://github.com/Catchxu/ACsleuth

Summary updated

Abstract

Fined-grained anomalous cell detection from affected tissues is critical for clinical diagnosis and pathological research. Single-cell sequencing data provide unprecedented opportunities for this task. However, current anomaly detection methods struggle to handle domain shifts prevalent in multi-sample and multi-domain single-cell sequencing data, leading to suboptimal performance. Moreover, these methods fall short of distinguishing anomalous cells into pathologically distinct subtypes. In response, we propose ACSleuth, a novel, reconstruction deviation-guided generative framework that integrates the detection, domain adaptation, and fine-grained annotating of anomalous cells into a methodologically cohesive workflow. Notably, we present the first theoretical analysis of using reconstruction deviations output by generative models for anomaly detection in lieu of domain shifts. This analysis informs us to develop a novel and superior maximum mean discrepancy-based anomaly scorer in ACSleuth. Extensive benchmarks over various single-cell data and other types of tabular data demonstrate ACSleuth's superiority over the state-of-the-art methods in identifying and subtyping anomalies in multi-sample and multi-domain contexts. Our code is available at https://github.com/Catchxu/ACsleuth.
BibTeX
@inproceedings{xu2024domain,
  author = {Kaichen Xu and Yueyang Ding and Suyang Hou and Weiqiang Zhan and Nisang Chen and Jun Wang and Xiaobo Sun},
  title = {{Domain Adaptive and Fine-grained Anomaly Detection for Single-cell Sequencing Data and Beyond}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure