Non-Parametric Feature Retrieval for Anomaly Detection

This direction is split into other directions: non-parametric distance-based detection. Its last version stays readable and citable here.

Version 1 of 168 papers75 conceptsfrozen 2026-10-04

In the quest to identify defects in industrial parts or rare diseases in medical scans, the dominant paradigm was once generative: build a model that learns to reconstruct "normal" data, and flag anything it fails to reconstruct as an anomaly. However, a growing body of research suggests that we might be overthinking the problem. Instead of training complex models to approximate the distribution of normal data, we can simply use the data itself as the model.

This shift toward non-parametric feature retrieval treats anomaly detection as a search problem. By storing representative features of normal samples in a "memory bank" and using nearest-neighbor search to score new inputs, researchers are achieving state-of-the-art results with significantly less training.

The Intuition of the Feature Space

The effectiveness of this approach relies on a simple observation: modern neural networks, even those not trained on your specific task, are excellent at mapping similar objects to nearby points in a high-dimensional feature space.

When we pass an image through a "frozen" foundation model like DINOv2 or CLIP, the resulting feature vectors (or embeddings) capture the essence of the object. In this space, "normal" samples naturally cluster together. An anomaly—a scratch on a metal surface or a tumor in an X-ray—will produce a feature vector that lands in a "lonely" part of the space, far from the cluster of known normal points.

As Yiyou Sun and colleagues demonstrated in KNN-OOD, this non-parametric approach avoids the "rigid statistical assumptions" of parametric models. While a Gaussian model might fail if the data distribution is complex or multi-modal, a K-Nearest Neighbor (K-NN) search simply looks at the local density, making it far more robust to the "messy" distributions found in real-world data.

The Memory Bank and the Coreset

Storing every single feature vector from a massive training set is computationally expensive. To make these systems deployable, researchers use a memory bank—a curated gallery of "normalcy."

A key breakthrough in this area is PatchCore, introduced by Karsten Roth and his team in Towards Total Recall in Industrial Anomaly Detection. Instead of storing entire images, PatchCore stores mid-level patch features. To keep the memory bank lean, they use "coreset sampling"—a technique that selects a small, diverse subset of features that effectively spans the same space as the full dataset.

During inference, the system breaks a new image into patches, extracts their features, and asks: "How far is this patch from its closest neighbor in our memory bank?" If the distance exceeds a threshold, the patch is flagged as a defect. This mechanism allows for precise pixel-level localization without ever having seen a "bad" example during training.

Why Retrieval Beats Reconstruction

Traditional reconstruction-based models (like VAEs or GANs) suffer from a "fidelity-stability dilemma." If the model is too powerful, it might accidentally learn to reconstruct the anomalies themselves, causing them to "disappear" from the error map.

Non-parametric retrieval bypasses this entirely. As Xingwu Zhang argues in Retrieval-based Anomaly Detection (RAD), by using a frozen backbone and a fixed memory bank, the "normal" reference remains stable. The anomaly score is a direct measure of distance to a known-good exemplar, providing a more reliable signal than a model's failure to reconstruct a pixel.

This "training-free" nature makes the approach ideal for few-shot learning. In AnomalyDINO, Mike Laszkiewicz and Asja Fischer show that by leveraging the inherent discriminative power of DINOv2, one can build a world-class anomaly detector using as little as a single "normal" reference image.

Beyond Images: Tabular and 3D Data

The principle of retrieval-based detection is not limited to 2D images. It is proving equally potent in other domains:

  • 3D Detection: In Back to the Feature (BTF), Eliahu Horwitz demonstrated that combining classical 3D features with a K-NN memory bank outperforms complex 3D deep learning models.
  • Tabular Data: Hugo Thimonier introduced NPT-AD, which uses Non-Parametric Transformers to model relationships between samples. By keeping the training set as a reference during inference, the model avoids the "distributional information loss" that occurs when a dataset is compressed into fixed model parameters.
  • Audio: Researchers like Manuel Barusco are even adapting these visual algorithms to sound, treating spectrograms as images and using memory banks to identify acoustic anomalies in industrial machinery.

The Path Forward

While non-parametric methods are highly effective, they are not without challenges. The reliance on a static memory bank can introduce memory overhead, a problem Hanxi Li addresses with the SNARM architecture by using "self-referential" matching within a single image. Furthermore, as Akib Mohammed Khan notes, these models can be vulnerable to adversarial attacks that shift features just enough to hide from the nearest neighbor.

Despite these hurdles, the move toward retrieval-based detection represents a significant simplification of the anomaly detection pipeline. By trusting the representational power of foundation models and the geometric intuition of nearest neighbors, researchers are building detectors that are faster to deploy, easier to explain, and more robust to the complexities of the real world.

Papers in this version

Showing 60 of 68 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_non_parametric_feature_retrieval_for_ano_0d0abb98,
  title = {Non-Parametric Feature Retrieval for Anomaly Detection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/non-parametric-feature-retrieval-for-anomaly-detection/rdr_0d0abb98f51147888704ee5b5c9fbcba},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_non_parametric_feature_retrieval_for_ano_0d0abb98_v1,
  title = {Non-Parametric Feature Retrieval for Anomaly Detection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/non-parametric-feature-retrieval-for-anomaly-detection/rdr_0d0abb98f51147888704ee5b5c9fbcba/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}