Object Centric Anomaly Segmentation

38 concepts21 papers1 proposal

Traditional anomaly and out-of-distribution (OOD) detection methods treat images as grids of independent pixels. When applied to semantic segmentation, these models output dense, per-pixel uncertainty heatmaps. However, as Shyam Nandan et al. point out in Unmasking Anomalies in Road-Scene Segmentation, this dense classification paradigm suffers from severe limitations: it generates high-frequency spatial noise, blurry boundaries, and isolated false positives where harmless background textures or shadows are misidentified as hazards.

Furthermore, pixel-wise approaches lack semantic cohesion. Hung Vu et al. observe in Robust Anomaly Detection in Videos Using Multilevel Representations that low-level detection methods struggle to identify an anomalous object as a single, unified entity, instead producing fragmented results based on local pixel variations. For downstream systems—such as autonomous vehicles that need to execute tactical maneuvers like braking or swerving—fuzzy heatmaps are insufficient. As Alexey Nekrasov 0001 et al. argue in UGainS: Uncertainty Guided Anomaly Instance Segmentation, autonomous systems require discrete object boundaries to treat anomalies as individual obstacles, yet pixel-wise methods frequently fail to distinguish overlapping or occluded instances.

The Core Intuition: Objectness as a Denoising Filter

To overcome these limitations, a powerful paradigm shift has emerged: object-centric and mask-level anomaly detection. The core intuition is simple yet profound: physical anomalies are almost always cohesive, structured objects, not isolated pixels. By leveraging the inherent "objectness" of physical entities, models can suppress stochastic noise and drastically reduce false positives.

Instead of asking "Is this pixel an anomaly?", these methods first group pixels into structurally coherent regions or masks, and then ask "Is this entire object an anomaly?". As Matej Grcic et al. demonstrate in On Advantages of Mask-level Recognition for Outlier-aware Segmentation, transitioning to mask-level recognition forces a consensus decision across a spatial region. This regional aggregation naturally filters out the speckle noise and boundary ambiguities common in pixel-wise scoring.

Pixel-Level Approach (Noisy & Fragmented):
[Pixel Anomaly Score] ──> [High-Frequency Noise] ──> Fragmented Heatmap

Object-Centric Approach (Clean & Cohesive):
[Pixels] ──> [Class-Agnostic Mask/Slot] ──> [Spatial Consensus Aggregation] ──> Unified Object Mask

By evaluating uncertainty at the object segment level, as originally proposed by Philipp Oberdiek et al. in Detection and Retrieval of Out-of-Distribution Objects in Semantic Segmentation, the system produces a significantly cleaner signal. If a network groups pixels into a cohesive mask but remains uncertain about its semantic label, the entire object-shaped region is flagged as an anomaly. This mechanism, explored by Jan Ackermann et al. in Maskomaly: Zero-Shot Mask Anomaly Segmentation, leverages the spatial consistency of mask-based architectures to treat entire predicted regions as single units for anomaly scoring, effectively filtering out background noise.

Architectural Mechanisms for Mask-Level Reasoning

Modern architectures implement this object-centric view through several distinct mechanisms:

Class-Agnostic Object Proposals

One approach uses a class-agnostic detector to group pixels into discrete object masks before computing anomaly scores. Victor Besnier et al. introduce an instance-aware pipeline in Instance-Aware Observer Network for Out-of-Distribution Object Segmentation that assigns a single anomaly score per object mask, averaging out localized errors across the object's surface.

Similarly, Alon Zolfi et al. develop this concept in YolOOD: Utilizing Object Detection Concepts for Multi-Label Out-of-Distribution Detection by combining a class-agnostic "objectness" score with class-specific confidences. The objectness score acts as a gatekeeper, ensuring that out-of-distribution scores are derived only from actual object candidates while completely suppressing background noise.

Transformer-Based Mask Queries

Modern vision transformers naturally lend themselves to mask-level reasoning. Youssef Shoeb et al. explain in Out-of-Distribution Segmentation in Autonomous Driving: Problems and State of the Art that mask-level reasoning can utilize transformer queries to determine if an entire segment is an outlier, capturing the structural integrity of hazards. Nikos Efthymiadis et al. confirm in A Dataset for Semantic Segmentation in the Presence of Unknowns that architectures utilizing mask-level predictions are inherently more effective at identifying unknown classes than those relying on pixel-level classification.

Foundation Model Prompting

With the rise of vision foundation models, researchers have begun using pixel-level uncertainty maps as "prompts" for powerful segmentation models like the Segment Anything Model (SAM). Mi Zheng demonstrates in Segmenting Objectiveness and Task-awareness Unknown Region for Autonomous Driving that integrating pixel-wise anomaly scores with pre-trained object-aware features allows for the segmentation of entire contiguous obstacles rather than fragmented parts. In zero-shot industrial settings, SoYoung Park et al. show in Zero-Shot Industrial Anomaly Segmentation with Image-Aware Prompt Generation that precise segmentation masks for anomalous regions can be generated by guiding a pre-trained segmentation model with bounding boxes identified by a vision-language model.

Object-Centric Latent Spaces

Beyond explicit masks, anomaly detection can be performed on object-level representations. Yin Xu et al. introduce OpenSlot: Mixed Open-set Recognition with Object-centric Learning, which decomposes images into independent semantic "slots." This modularity prevents high-confidence signals from known objects from masking co-occurring unknown objects.

Furthermore, Max Peter Ronecker et al. show in Vision Foundation Model Embedding-Based Semantic Anomaly Detection that analyzing vision foundation model embeddings at the object level rather than on a fixed grid filters out irrelevant visual noise and focuses on the identity and context of specific entities.

Eliminating Background Clutter via Spatial Grounding

A major benefit of object-centric anomaly detection is its ability to isolate the target of interest from complex backgrounds, which are frequent sources of false positives.

In industrial defect detection, Y.F. Cai et al. show in FiLo: Zero-Shot Anomaly Detection by Fine-Grained Description and High-Quality Localization that a preliminary grounding step to identify the target object and its constituent parts effectively eliminates false positives in background textures or shadows. This spatial constraint ensures the vision model only evaluates relevant object surfaces. Similarly, Samet Hiçsönmez et al. integrate unsupervised segmentation into a diffusion-based inversion pipeline in Training Free Zero-Shot Visual Anomaly Localization via Diffusion Inversion to focus exclusively on object-centric regions, masking out background clutter that might contain irrelevant irregularities.

In video surveillance and autonomous driving, J. Marius Zoellner et al. demonstrate in Hybrid Video Anomaly Detection for Autonomous Driving that generating dense scores localized within object bounding boxes pinpoints specific agents responsible for anomalous behavior while ignoring irrelevant background noise. Noël Ollick et al. extend this in UMAD: Unsupervised Mask-Level Anomaly Detection for Autonomous Driving, showing that aggregating anomaly scores within unsupervised object masks filters out high-frequency reconstruction noise and artifacts by grounding the detection in discrete, semantically coherent units.

Beyond Obstacles: Identifying Structural Anomalies

The utility of object-centric anomaly detection extends beyond physical obstacles to abstract structural failures. For instance, in evaluating text-to-image generative models, identifying physical hallucinations (like extra limbs or warped features) is traditionally challenging for pixel-wise regression.

Shuhao Han et al. show in NTIRE 2025 challenge on Text to Image Generation Model Quality Assessment that treating these structural anomalies as an instance segmentation task—identifying and masking specific regions of distortion as discrete objects—leads to significantly higher detection accuracy. By framing structural distortions as anomalous object instances rather than dense classification errors, the model leverages spatial cohesion to accurately localize and categorize complex generative failures.

Key Foundational Work

Detection and Retrieval of Out-of-Distribution Objects in Semantic Segmentation (2020) — contributed 3 concept elements to this direction.
Robust Anomaly Detection in Videos Using Multilevel Representations (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_segment_based_anomaly_detection_4eaec6ac,
  title = {segment-based anomaly detection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/segment-based-anomaly-detection/rdr_4eaec6ac47c24e60b6ffb47abed8529e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_segment_based_anomaly_detection_4eaec6ac_v1,
  title = {segment-based anomaly detection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/segment-based-anomaly-detection/rdr_4eaec6ac47c24e60b6ffb47abed8529e/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake