Domain-Agnostic Outlier Ranking Algorithms—A Configurable Pipeline for Facilitating Outlier Detection in Scientific Datasets

Hannah Kerner, Umaa Rebbapragada, Kiri L. Wagstaff, Steven Lu, Bryce Dubayah, Eric Huff, Jake Lee, Vinay Raman, Sakshum KulshresthaFrontiers in Astronomy and Space Sciences 2022 · DOI

Abstract

Automatic detection of outliers is universally needed when working with scientific datasets, e.g., for cleaning datasets or flagging novel samples to guide instrument acquisition or scientific analysis. We present Domain-agnostic Outlier Ranking Algorithms (DORA), a configurable pipeline that facilitates application and evaluation of outlier detection methods in a variety of domains. DORA allows users to configure experiments by specifying the location of their dataset(s), the input data type, feature extraction methods, and which algorithms should be applied. DORA supports image, raster, time series, or feature vector input data types and outlier detection methods that include Isolation Forest, DEMUD, PCA, RX detector, Local RX, negative sampling, and probabilistic autoencoder. Each algorithm assigns an outlier score to each data sample. DORA provides results interpretation modules to help users process the results, including sorting samples by outlier score, evaluating the fraction of known outliers in n selections, clustering groups of similar outliers together, and web visualization. We demonstrated how DORA facilitates application, evaluation, and interpretation of outlier detection methods by performing experiments for three real-world datasets from Earth science, planetary science, and astrophysics, as well as one benchmark dataset (MNIST/Fashion-MNIST). We found that no single algorithm performed best across all datasets, underscoring the need for a tool that enables comparison of multiple algorithms.

Introduction

In the pursuit of scientific discovery, identifying anomalous data points—the "needles in the haystack"—is a fundamental task. Whether the goal is to remove erroneous sensor readings or to identify rare astronomical phenomena, outlier detection is critical for both data quality and novel discovery. However, the adoption of state-of-the-art machine learning (ML) techniques for outlier detection is often hindered by technical complexity and the "no free lunch" theorem: no single algorithm is universally superior across all scientific domains.

To address these challenges, we introduce DORA (Domain-agnostic Outlier Ranking Algorithms), an open-source, configurable pipeline designed to simplify the application and evaluation of outlier detection methods. DORA provides a unified framework that allows researchers to apply multiple algorithms to diverse data types with minimal coding overhead.

The DORA Framework

DORA is implemented as a modular, object-oriented pipeline in Python. Its architecture is designed to handle the end-to-end process of outlier detection, from data ingestion to result visualization.

The DORA pipeline architecture diagram, illustrating the flow from configuration and data loading to visualization.

The pipeline consists of four primary stages:

  1. Data Loading: DORA supports a wide range of input formats, including time series, feature vectors, grayscale/RGB images, and multi-band rasters common in remote sensing.
  2. Outlier Ranking: The core of DORA includes seven unsupervised algorithms: Reconstruction-based (PCA, DEMUD), Distance-based (RX, Local RX), Sparsity-based (Isolation Forest), and Probability-based (Negative Sampling, Probabilistic Autoencoder).
  3. Interpretation: To make sense of the rankings, DORA includes modules for clustering results (using K-means or Self-Organizing Maps) and evaluating performance against known ground-truth labels.
  4. Visualization: A dedicated desktop application allows for side-by-side comparison of algorithm results, facilitating rapid qualitative assessment.

Experimental Evaluation

We evaluated DORA's versatility across four distinct scientific domains to demonstrate its broad applicability:

  • Astrophysics: 100,000 galaxy objects from the Dark Energy Survey (DES).
  • Planetary Science: 1,050 rock targets from Mars Science Laboratory (MSL) rover Navcam images.
  • Earth Science: 6,757 Sentinel-1 radar time series for identifying mislabeled crop fields.
  • Benchmark: A combination of MNIST (outliers) and Fashion-MNIST (inliers).

Results and Discussion

Our experiments confirmed that algorithm performance is highly domain-dependent. For instance, PCA achieved the highest precision in the astrophysics dataset (0.42), while Isolation Forest performed best for planetary images (0.56). In the Earth science domain, Negative Sampling was the top performer (0.49).

The DORA Visualizer interface, enabling side-by-side qualitative comparison of algorithm rankings.

The DORA Visualizer proved essential for interpreting these results. By allowing researchers to view the top-ranked outliers from different algorithms simultaneously, it becomes easier to identify which methods are capturing meaningful scientific anomalies versus those flagging data artifacts.

A key insight from our work is the value of DEMUD (Discovery through Eigenbasis Modeling of Uninteresting Data). Unlike traditional methods that may rank many similar outliers highly, DEMUD prioritizes diversity by deprioritizing samples similar to those already selected, making it particularly effective for rapid discovery of novel classes.

Conclusion

DORA lowers the barrier to entry for advanced outlier detection in the sciences. By providing a configurable, multi-algorithm pipeline, it enables researchers to move beyond "default" choices and find the best-performing methods for their specific data. Future work will focus on integrating causal inference to improve interpretability and deploying DORA within active mission workflows for real-time discovery.

Code Availability: DORA is open-source and available on GitHub at github.com/nasaharvest/dora.

Summary updated

Abstract

Automatic detection of outliers is universally needed when working with scientific datasets, e.g., for cleaning datasets or flagging novel samples to guide instrument acquisition or scientific analysis. We present Domain-agnostic Outlier Ranking Algorithms (DORA), a configurable pipeline that facilitates application and evaluation of outlier detection methods in a variety of domains. DORA allows users to configure experiments by specifying the location of their dataset(s), the input data type, feature extraction methods, and which algorithms should be applied. DORA supports image, raster, time series, or feature vector input data types and outlier detection methods that include Isolation Forest, DEMUD, PCA, RX detector, Local RX, negative sampling, and probabilistic autoencoder. Each algorithm assigns an outlier score to each data sample. DORA provides results interpretation modules to help users process the results, including sorting samples by outlier score, evaluating the fraction of known outliers in n selections, clustering groups of similar outliers together, and web visualization. We demonstrated how DORA facilitates application, evaluation, and interpretation of outlier detection methods by performing experiments for three real-world datasets from Earth science, planetary science, and astrophysics, as well as one benchmark dataset (MNIST/Fashion-MNIST). We found that no single algorithm performed best across all datasets, underscoring the need for a tool that enables comparison of multiple algorithms.
BibTeX
@inproceedings{kerner2022domain,
  author = {Hannah Kerner and Umaa Rebbapragada and Kiri L. Wagstaff and Steven Lu and Bryce Dubayah and Eric Huff and Jake Lee and Vinay Raman and Sakshum Kulshrestha},
  title = {{Domain-Agnostic Outlier Ranking Algorithms—A Configurable Pipeline for Facilitating Outlier Detection in Scientific Datasets}},
  booktitle = {Frontiers in Astronomy and Space Sciences},
  year = {2022},
  doi = {10.3389/fspas.2022.867947},
}
Figure