Guiding Scientific Discovery with Explanations Using DEMUD

Kiri L. Wagstaff, Nina L. Lanza, David R. Thompson, Thomas G. Dietterich, Martha S. GilmoreAAAI 2013 · DOI

Introduction

In the era of "big data," scientists face an overwhelming volume of information. From hyperspectral Martian imagery to forensic glass samples, the sheer scale of data precludes manual review. While automated anomaly detection offers a solution, traditional "black box" algorithms often suffer from high redundancy—flagging numerous similar anomalies—and a lack of interpretability, failing to explain why a specific data point is considered interesting.

To address these challenges, we present DEMUD (Discovery through Eigenbasis Modeling of Uninteresting Data). DEMUD is an iterative machine learning algorithm designed to help researchers find rare or novel items while providing domain-specific explanations and updating its internal model in real-time to prioritize diversity in discovery.

Methodology

The core philosophy of DEMUD is to model what is "uninteresting" rather than what is "interesting." By defining the known or common data, the algorithm can more effectively highlight the unknown.

  • Eigenbasis Modeling: DEMUD uses Singular Value Decomposition (SVD) or Principal Component Analysis (PCA) to create a low-dimensional "eigenbasis" representing the data it has already seen or that is prevalent in the set.
  • Selection via Reconstruction Error: New data points are projected onto this basis. Points with a high reconstruction error—those the model cannot explain—are flagged as potentially novel.
  • Incremental Updates: After a selection is reviewed, DEMUD updates its model to include that item. This ensures subsequent selections are distinct from previous ones, maximizing the diversity of discovered classes.
  • Per-Feature Explanations: DEMUD generates explanations by analyzing residuals—the specific features (e.g., chemical elements or wavelengths) that the model failed to reconstruct.

Experimental Evaluation

DEMUD was evaluated across three distinct domains: forensic glass analysis (UCI Glass Dataset), orbital mineralogy (CRISM Hyperspectral Data), and Martian surface chemistry (ChemCam Point Spectra).

Benchmark Performance

On the UCI Glass dataset, which contains six types of glass, DEMUD demonstrated superior efficiency compared to existing state-of-the-art methods.

DEMUD discovers all 6 glass classes significantly faster than competing methods like CLOVER.

As shown in Figure 1, DEMUD identified all six classes in just 8 selections, whereas the previous best method, CLOVER, required 15.

Finding Rare Targets

In the CRISM Mars data, the goal was to locate rare magnesite deposits, which constitute only 0.06% of the data. DEMUD successfully identified the magnesite in just 5 selections. In contrast, a static SVD approach—which does not update its model iteratively—required over 200 selections to find the same target.

Enhancing Human Accuracy

A critical finding of this research is that providing residuals as explanations significantly improves human decision-making. In studies with planetary scientists using ChemCam spectra, human classification accuracy doubled when DEMUD’s explanations were provided.

Left: ChemCam spectra with DEMUD-generated explanations highlighting key elemental lines. Right: Cumulative accuracy showing the performance boost provided by these explanations.

As illustrated in Figure 6, accuracy for the first 10 selections reached as high as 90% with explanations, compared to roughly 25% without them. This demonstrates that interpretability is not merely a convenience but a functional requirement for effective scientific discovery.

Conclusion and Future Work

DEMUD offers a scalable and interpretable framework for navigating massive scientific datasets. By focusing on modeling the "uninteresting" and providing feature-level explanations, it bridges the gap between automated detection and human expertise. Future work will explore the integration of "robust incremental SVD" to better handle noisy data and the ability to seed the initial model with specific prior knowledge to further streamline the discovery process.

Summary updated

BibTeX
@inproceedings{wagstaff2013guiding,
  author = {Kiri L. Wagstaff and Nina L. Lanza and David R. Thompson and Thomas G. Dietterich and Martha S. Gilmore},
  title = {{Guiding Scientific Discovery with Explanations Using DEMUD}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2013},
  doi = {10.1609/aaai.v27i1.8561},
}
Figure