Identifying and Benchmarking Natural Out-of-Context Prediction Problems

David Madras, Richard Zemel

Introduction

Deep learning vision models frequently rely on "shortcuts"—learning, for instance, that cows typically appear on green grass, and subsequently failing to recognize a cow on a sandy beach. To build safer and more reliable systems, it is crucial to benchmark how well models handle these Out-Of-Context (OOC) examples.

However, defining what makes an image "out-of-context" is highly subjective. Consequently, existing benchmarks often rely on manual curation or semi-synthetic data (e.g., pasting a cutout of a bird onto a forest background). The field has lacked a scalable, principled way to identify naturally occurring OOC examples to rigorously test how models handle complex, real-world context shifts.

This paper introduces a framework to automatically mine natural OOC examples from existing datasets using auxiliary annotations, culminating in a new benchmark called NOOCH (Naturally-Occurring Out-of-context Challenge sets). By evaluating robust learning algorithms on NOOCH, the authors reveal that methods excelling on synthetic OOC datasets often struggle significantly more on natural, complex context shifts.

Methodology: Defining Out-of-Context

The authors propose a unifying framework that leverages auxiliary dataset annotations—such as segmentation masks or text captions—to programmatically define an OOC criterion. They instantiate this framework in two distinct ways to define "hard positives" (the object is present, but its usual context is missing) and "hard negatives" (the object is absent, but its usual context is present):

  1. Co-occurrence/Extractibility (CE) Criterion: This method uses segmentation areas to find objects that usually appear together (e.g., cars and roads). A "hard positive" for a car would be an image containing a car, but where the road takes up very little or no space.
  2. Gist Criterion: This method uses SBERT embeddings of image captions to capture the overall semantic "vibe" or "gist" of a scene. Hard examples are those where the object's presence or absence contradicts the typical semantic gist of scenes where that object is normally found.
Examples of Hard Positive and Hard Negative images from the NOOCH dataset.

The NOOCH Benchmark

The base datasets used to build the benchmark are COCO and COCO-Stuff. COCO-Stuff is particularly crucial because it provides segmentation masks for background "stuff" (like sky, grass, and road), which act as the context.

From this, the authors introduce NOOCH, featuring 12 binary object presence tasks (e.g., car, bowl, cow). NOOCH is split into NOOCH-CE and NOOCH-Gist based on the two criteria described above. The authors also utilize the standard semi-synthetic Waterbirds dataset as a baseline to compare algorithm performance.

Key Insights and Results

The authors evaluated standard Expected Risk Minimization (ERM) against several robust learning methods, including Label-based adjustments (Reweighting), Environment-based methods (Group DRO, IRM), and Adaptive methods (CVaR, Focal Loss).

  • Synthetic vs. Natural Benchmarks Tell Different Stories: Algorithms like Group DRO appear highly effective on semi-synthetic datasets like Waterbirds, beating standard ERM by a massive ~0.3 in worst-group error. However, they show much more modest gains on natural datasets. On NOOCH-CE, ERM achieves an AUC of 0.865 on hard examples, while Group DRO achieves 0.911. This suggests semi-synthetic datasets might overestimate our current progress in robust machine learning.
  • The "Gist" is Harder: Models struggle more with holistic semantic shifts (NOOCH-Gist) than with the absence of specific co-occurring objects (NOOCH-CE). On NOOCH-Gist, the performance gap is even smaller: ERM gets 0.826 AUC, and Group DRO gets 0.849. Holistic scene understanding remains a major weakness.
  • Data Over Algorithm: Simple data reweighting based on environment labels performs just as well as complex, specialized algorithms like Group DRO or IRM. Having the right auxiliary data to define the environment is more critical than the specific mathematics of the robust optimizer.
  • Focal Loss Surprise: Focal loss—an adaptive method that dynamically upweights high-loss examples and requires no environment labels—performed surprisingly well. It matched the environment-based methods on hard examples while maintaining a strong overall AUC, offering a highly practical tradeoff.
  • The Robustness Tradeoff: Improving performance on hard (OOC) examples almost always degrades performance on easy (in-context) examples. There is no free lunch.

Limitations and Future Work

The authors acknowledge that automated OOC discovery is not perfect. It relies on proxies like caption embeddings and will inevitably miss nuanced OOC cases that require human intuition to spot. Furthermore, the NOOCH benchmark is currently restricted to binary classification (presence vs. absence of an object), which is simpler than real-world multi-class or dense segmentation tasks.

Looking forward, the authors suggest using this automated, annotation-driven approach to rapidly generate new challenge sets. By treating auxiliary data as a tool for "automatic test generation," researchers can scalably build unit tests for deep learning models to uncover specific, natural failure modes before deployment.

Summary updated

Abstract

Deep learning systems frequently fail at out-of-context (OOC) prediction, the problem of making reliable predictions on uncommon or unusual inputs or subgroups of the training distribution. To this end, a number of benchmarks for measuring OOC performance have been recently introduced. In this work, we introduce a framework unifying the literature on OOC performance measurement, and demonstrate how rich auxiliary information can be leveraged to identify candidate sets of OOC examples in existing datasets. We present NOOCh: a suite of naturally-occurring "challenge sets", and show how varying notions of context can be used to probe specific OOC failure modes. Experimentally, we explore the tradeoffs between various learning approaches on these challenge sets and demonstrate how the choices made in designing OOC benchmarks can yield varying conclusions.
BibTeX
@inproceedings{madras2021identifying,
  author = {David Madras and Richard Zemel},
  title = {{Identifying and Benchmarking Natural Out-of-Context Prediction Problems}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=ByPR_hOE_EY},
}
Figure