How Far Is Too Far? Object Recognition Declines Monotonically with Semantic Distance

Domenic Bersch, Hai Van Tran, Gemma RoigarXiv 2026 · arXiv

TL;DR

Vision models don’t just fail in "weird" contexts; their performance drops predictably and monotonically as the semantic distance between an object and its background increases. Researchers introduced the ImageNet-OOC1k benchmark to prove this and discovered "aggregation failure" in Vision Transformers—where the model correctly identifies an object at the patch level but fails to consolidate that information into its final prediction.

What Problem Does This Paper Address?

Computer vision models are notorious for relying on "shortcuts," such as recognizing a cow because it’s standing in a green field rather than looking at the cow itself. While we know that incongruent backgrounds (like a cow in a kitchen) hurt accuracy, existing research usually treats context as a binary: either it's "right" or "wrong."

This paper asks a more nuanced question: How does recognition change as the mismatch between object and background grows progressively larger? To answer this, the authors needed a way to measure the "semantic distance" between a background (e.g., a forest) and an object (e.g., a toaster) on a continuous scale.

Methods

The authors developed a two-part approach to quantify and test context sensitivity:

  1. The Consensus Visual-Semantic Axis: To define "distance," they looked at the final linear layers (classifier heads) of 10 different pretrained models. By treating these weights as "class prototypes" in a high-dimensional feature space and calculating their cosine dissimilarity, they created a model-agnostic map of how semantically related different ImageNet classes are.
  2. Controlled Composite Generation: They used ImageNet-S segmentation masks to extract objects and IOPaint to create "empty" backgrounds. They then systematically placed objects into backgrounds at specific semantic distances (e.g., placing a "jacamar" bird into a "television" background) to see exactly when and how the models failed.
  3. Mechanistic Analysis (Logit Lens): To see why Vision Transformers (ViTs) fail, they used the "logit lens" technique. This allowed them to inspect individual spatial patches within the model to see if the object evidence was present internally, even if the final output was wrong.

Datasets

  • ImageNet-OOC1k (New): A benchmark featuring 1,137 background-only scenes and a library of 12,321 object instances across 918 classes. It allows for ~14 million possible object-background combinations.
  • ImageNet-S: Used for high-quality segmentation masks.
  • ImageNet: The source for the underlying classes and images.

Key Insights and Ideas

  • The Monotonic Decline: Across 35 different models (CNNs, ViTs, and CLIP), accuracy didn't just drop—it fell in a remarkably consistent, smooth curve as semantic distance increased. This suggests context sensitivity is a fundamental, predictable property of modern vision systems.
  • Backgrounds are "Soft" Classifiers: Even when the object is completely removed, models still predict classes within the correct semantic neighborhood of the original background about 25% of the time. The background provides a "prior" that the model has to overcome.
  • Aggregation Failure: The most striking finding is the "Found-but-Lost" (FBL) error. In many cases, the ViT's internal patches correctly identify the object, but the global "CLS token" (which makes the final call) fails to aggregate that evidence, instead getting "pulled" toward a background-related prediction at the very last layer.
  • Spatial Compactness as a Signal: Object evidence tends to be spatially concentrated (a tight cluster of patches), while background noise is more dispersed. This difference in "spatial signature" can be used to diagnose and even fix errors.

Key Results

  • Performance Drop: Top-1 accuracy decreased by 9 to 19 percentage points as pairings moved from semantically similar to highly dissimilar.
  • Error Recovery: By using a simple "spatial re-ranking" probe—which looks for compact clusters of evidence in intermediate layers to override the final prediction—the authors recovered a significant portion of FBL errors.
  • Accuracy Gain: This re-ranking strategy improved in-sample accuracy by up to 5.7 percentage points on a ViT-B/16, with a high "recovery-to-breakage" ratio of 9.4:1 (meaning for every ~9 errors fixed, only 1 correct prediction was accidentally broken).

Limitations and Future Work

  • Compositing Artifacts: While the authors controlled for factors like object size and edge discontinuities, synthetic composites can never perfectly replicate the lighting and physics of natural scenes.
  • Single-Object Focus: The study focuses on one object in one background. Real-world scenes are multi-object, which adds layers of contextual complexity.
  • Diagnostic vs. Deployable: The re-ranking method is currently a "diagnostic probe" that requires access to the model's internal layers. Turning this into a plug-and-play inference method for any model remains a challenge.
  • Facilitation vs. Interference: The paper notes that some backgrounds actually help recognition as they get further away (facilitation), while others hinder it (interference). The exact reason for this split—likely related to scene diversity—requires further study.

Summary updated

Abstract

Vision models are sensitive to object-background context, yet it remains unclear how recognition changes as the semantic distance between objects and backgrounds increases. We introduce ImageNet-OOC1k, a benchmark enabling graded control of object-background relationships along a continuous consensus visual-semantic dissimilarity axis. This formulation reveals a highly consistent monotonic decline in recognition accuracy across 35 models spanning convolutional neural networks, Vision Transformers, and vision-language models. Internal analyses of Vision Transformers reveal patterns consistent with aggregation failure, where object evidence is present at the patch level but not consolidated into the final prediction. Using this characterization as a diagnostic probe, a simple spatial re-ranking strategy recovers a subset of these errors and improves in-sample accuracy by up to 5.7 percentage points. These results establish context sensitivity as a continuous and predictable property of modern vision systems, with Vision Transformer analyses suggesting that aggregation limitations may contribute to some failure types even when object evidence remains available.
BibTeX
@misc{bersch2026how,
  author = {Domenic Bersch and Hai Van Tran and Gemma Roig},
  title = {{How Far Is Too Far? Object Recognition Declines Monotonically with Semantic Distance}},
  year = {2026},
  eprint = {2611.19747},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.19747},
}
Figure