Toward a Taxonomy and Computational Models of Abnormalities in Images

Babak Saleh, Ahmed Elgammal, Jacob Feldman, Ali Farhadi

This paper explores a fundamental yet overlooked capability of the human visual system: the ability to instantly recognize when an image is "weird" and explain why. While state-of-the-art computer vision models excel at identifying typical objects, they often stumble when faced with atypical instances. The authors propose a formal taxonomy of visual abnormalities, a new dataset for benchmarking, and a computational framework that detects "surprising" images by modeling what is normal.

TL;DR

The authors define a three-part taxonomy of visual abnormality—Object-centric, Context-centric, and Scene-centric—and introduce a generative model that detects atypicality by measuring "surprise" relative to normal data. Their approach outperforms prior methods and highlights how standard deep learning models fail to generalize to the "weird" corners of the visual world.

What Problem Does This Paper Address?

Most computer vision algorithms are trained on "typical" examples. However, the real world is full of atypicality: a car made of wood, a boat in a living room, or a street engulfed in flames.

Current models lack the flexibility to handle these cases. For instance, the authors show that top-tier Convolutional Neural Networks (CNNs) like VGG-19, which have low error rates on standard datasets, see their error rates nearly triple when tested on abnormal images (see Figure 5). The challenge is that "abnormality" isn't a single thing; it’s a complex, graded phenomenon. A car can be abnormal because of its shape, its location, or the event it's involved in. This paper seeks to categorize these reasons and build a model that can reason about them without needing to see abnormal training examples.

Table 1: Standard CNNs fail to generalize to abnormal images.

Methods

The researchers used a two-pronged approach: human psychology and generative modeling.

  1. Human-Subject Experiments: They asked humans to describe why specific images looked strange. Using agglomerative clustering on these responses (Figure 2), they discovered a coarse taxonomy of three major categories:
  2. Object-centric: Issues with the object itself (e.g., weird shape, material, or missing parts).
  3. Context-centric: A normal object in a strange place or at a strange scale (e.g., an elephant in a room).
  4. Scene-centric: The entire scene is atypical (e.g., a strange event like a flood or fire).
  5. Computational Framework: Instead of training a classifier to recognize "abnormal" (which is hard because abnormality is infinite), they built a Bayesian generative model of normality.
  6. Modeling Typicality: They modeled the relationships between scenes, objects, and their semantic attributes (e.g., "wooden," "sunny").
  7. Surprise Scores: For a new image, the model calculates a "Surprise Score" ($I = -\log P$) for each category. If the probability of seeing a certain attribute in a given context is very low, the surprise score spikes.
  8. Normalization: Since surprise scores for different categories (like shape vs. location) aren't directly comparable, they mapped these scores to a common scale using Inverse Gaussian distributions fitted on normal data.
Figure 3: Graphical Model of Normal Images

Datasets

  • 1001 Abnormal Image Dataset: A new, comprehensive benchmark introduced by the authors. It includes 1,001 unique images showing a wide range of atypicalities, validated and annotated by human subjects.
  • Training Data: To learn what is "normal," the authors used standard datasets: PASCAL (for objects), SUN (for scenes), and PASCAL Context (for object-scene relationships).

Key Insights and Ideas

  • Abnormality is a Spectrum: There is no hard boundary between normal and abnormal. The authors treat it as a "graded phenomenon," which is why generative models (modeling the distribution of "normal") perform better than discriminative ones (trying to draw a line between normal and abnormal).
  • The Power of Attributes: Low-level features (like edges) might identify a car, but high-level attributes (like "furry") are what signal that the car is abnormal. The model's ability to reason about why something is strange relies heavily on these semantic descriptors.
  • Taxonomy Matters: By breaking abnormality into three distinct scores, the model can not only detect that an image is strange but also provide a reason (e.g., "this is contextually abnormal").

Key Results

The proposed model significantly outperformed prior state-of-the-art methods in abnormality recognition.

  • Object-centric detection: Achieved an AUC of 0.9311, beating the previous baseline of 0.9125.
  • Context-centric detection: Achieved an AUC of 0.8943, outperforming the previous best of 0.8518.
  • Reasoning: The model's predicted reasons for abnormality showed high correlation with human judgments. As seen in Figure 4, the surprise scores effectively separate normal images (purple stars) from abnormal ones, while also clustering different types of abnormalities.
Figure 4: Plotting images in 3-D based on three surprise score that we get from our computational models. Points are colored based on the most important reason of abnormality in each image. Colorful clouds of abnormal images are sep- arated from normal images (spread of purple stars close to the center of coordinate).

Limitations and Future Work

  • Bounding Box Dependency: The current model assumes that object bounding boxes are provided. Because standard object detectors often fail on abnormal objects, the system isn't yet fully autonomous.
  • Attribute Noise: The model relies on attribute classifiers, which can be noisy. Improving the reliability of these semantic detectors would directly improve abnormality detection.
  • Taxonomy Depth: The authors suggest that "digging deeper" into the sub-categories of their taxonomy (e.g., distinguishing between a "weird material" and a "missing part") is a promising direction for future research.

Summary updated

Abstract

The human visual system can spot an abnormal image, and reason about what makes it strange. This task has not received enough attention in computer vision. In this paper we study various types of atypicalities in images in a more comprehensive way than has been done before. We propose a new dataset of abnormal images showing a wide range of atypicalities. We design human subject experiments to discover a coarse taxonomy of the reasons for abnormality. Our experiments reveal three major categories of abnormality: object-centric, scene-centric, and contextual. Based on this taxonomy, we propose a comprehensive computational model that can predict all different types of abnormality in images and outperform prior arts in abnormality recognition.
BibTeX
@inproceedings{saleh2016toward,
  author = {Babak Saleh and Ahmed Elgammal and Jacob Feldman and Ali Farhadi},
  title = {{Toward a Taxonomy and Computational Models of Abnormalities in Images}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2016},
  doi = {10.1609/aaai.v30i1.10468},
}
Figure