Toward a Taxonomy and Computational Models of Abnormalities in Images

Babak Saleh, Ahmed M. Elgammal, Jacob Feldman, Ali Farhadi

Abstract

While humans possess an innate ability to identify "weird" or "abnormal" visual stimuli, computer vision models have historically struggled with anything outside narrow statistical norms. This paper addresses this gap by proposing a formal taxonomy of image abnormalities—categorized into object-centric, context-centric, and scene-centric types. By developing a Bayesian generative model that learns "typicality" rather than "abnormality," the authors created a system capable of detecting and reasoning about strange images, significantly outperforming standard deep learning models of the era.

Introduction: The Challenge of Visual Abnormalities

In 2015, state-of-the-art Convolutional Neural Networks (CNNs) were achieving record-breaking performance on standard datasets. However, these models were fundamentally "brittle." While they could identify a typical car with high accuracy, they often failed when presented with a car made of grass or a car perched on a rooftop.

The authors identified two primary hurdles in the field: a lack of a formal definition for what constitutes an "abnormal" image and a "generalization gap" where machines lacked a model of typicality to recognize deviations.

Figure 1: Examples of images ranging from less to more abnormal across different categories.

Methodology: Defining and Modeling the "Weird"

The researchers adopted a human-centric approach to define the problem space before developing their computational framework.

1. A Human-Derived Taxonomy

Through experiments with human subjects, the authors collected descriptions of why specific images appeared abnormal. This qualitative data was clustered to reveal three high-level categories:

  • Object-centric: Abnormalities inherent to the object itself, such as strange textures (e.g., a wooden car) or misplaced parts.
  • Context-centric: The object is normal, but its relationship to its environment is unexpected (e.g., a car floating in a lake).
  • Scene-centric: The entire scene depicts a strange event or possesses atypical global attributes.

2. Modeling Typicality via Surprise Scores

Rather than attempting to train a model on the infinite variety of "abnormal" images, the authors built a Bayesian generative model of normality. By learning the typical relationships between objects, their semantic attributes (color, texture), and their locations, the model can calculate "Surprise Scores."

When an image deviates from the learned normal distribution, the surprise score spikes. These scores are calculated for object, context, and scene levels and are normalized using an Inverse Gaussian distribution to make them comparable across different categories.

Key Results and Insights

The proposed model achieved an Area Under the Curve (AUC) of 0.89 for classifying normal versus abnormal images, a significant improvement over contemporary baselines.

Figure 5: 3D visualization of the model's ability to cluster images into object-centric, context-centric, and scene-centric abnormalities.

One of the most striking findings was the "blindness" of standard CNNs to these abnormalities. Top-tier models saw their error rates jump from ~30% on normal images to over 76% on abnormal ones. This suggests that discriminative training on standard datasets fails to capture the underlying essence of a category.

Furthermore, the model demonstrated an ability to "reason" about its detections. It could successfully identify which of the three taxonomic categories was the primary cause of the abnormality. Ablation studies highlighted that for context-centric issues, the location of the object was the most critical factor in triggering a high surprise score.

Discussion and Future Directions

While the framework represents a major step forward, several limitations remain. The current system relies on provided object bounding boxes, as standard detectors often fail to locate abnormal objects. Additionally, the study was limited to six object classes (Airplane, Boat, Car, Chair, Motorbike, Sofa), and scaling this to thousands of real-world categories remains a challenge. Finally, the model's performance is inherently tied to the reliability of its underlying attribute classifiers; noise in semantic detection directly impacts the accuracy of the surprise scores.

Despite these challenges, the paper provides a robust foundation for moving beyond simple classification toward a more nuanced, human-like understanding of visual typicality and abnormality.

Summary updated

Abstract

The human visual system can spot an abnormal image, and reason about what makes it strange. This task has not received enough attention in computer vision. In this paper we study various types of atypicalities in images in a more comprehensive way than has been done before. We propose a new dataset of abnormal images showing a wide range of atypicalities. We design human subject experiments to discover a coarse taxonomy of the reasons for abnormality. Our experiments reveal three major categories of abnormality: object-centric, scene-centric, and contextual. Based on this taxonomy, we propose a comprehensive computational model that can predict all different types of abnormality in images and outperform prior arts in abnormality recognition.
BibTeX
@inproceedings{saleh2016toward,
  author = {Babak Saleh and Ahmed M. Elgammal and Jacob Feldman and Ali Farhadi},
  title = {{Toward a Taxonomy and Computational Models of Abnormalities in Images}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2016},
  doi = {10.1609/aaai.v30i1.10468},
}
Figure