Object-Only Is Not Context-Free: A Counterfactual Diagnostic for Context Removal in Object Retrieval

Boaz Meivar, Shai AvidanarXiv 2026 · arXiv

In the world of computer vision, we often assume that if we have a perfect mask for an object, we can extract a "pure" representation of that object by simply ignoring everything outside the mask. A new paper from researchers at Tel Aviv University, titled "Object-Only Is Not Context-Free," reveals that this assumption is fundamentally flawed. Modern neural networks are so "leaky" that their object descriptors are deeply entangled with the surrounding scene, leading to significant failures when context is removed or changed.

TL;DR

The authors introduce the Keyhole Aperture Probe (KAP), a diagnostic tool that measures how much an object's feature representation relies on its background. They find that "object-only" descriptors are rarely context-free, often losing over half their accuracy when the background is removed, but they also provide a lightweight adapter that can significantly reduce this dependence and improve overall retrieval performance.

What Problem Does This Paper Address?

Object retrieval systems—like those used in image search or robotics—frequently use a "segment-then-pool" approach. They find an object mask, pool features from that region, and treat the rest of the image as noise.

However, modern foundation models (like ViTs and large CNNs) have massive receptive fields and global attention mechanisms. This means that the "object features" computed inside a mask are heavily influenced by pixels far outside of it. If a model learns to recognize a "donut" partly by the "table" it's sitting on, what happens when you try to retrieve that same donut in a different scene?

The authors point out that this context reliance creates a "shortcut" behavior. When you mask out the background, you aren't just removing noise; you're often removing the very signals the model has come to rely on, sometimes causing it to misidentify the object entirely.

Figure 1. How much context does an object descriptor need? A COCO donut query under an expanding visible aperture (gray fill), with DINOv2-G/14 top-1 retrievals. At d=0 (mask only) the top-1 is wrong (orange); as the aperture opens the match flips through cake to the correct donut. KAP reduces this dependence to one scalar; across ten backbones it varies widely and is not exclusive to ViT self-attention.

Methods

To quantify this effect, the researchers developed the Keyhole Aperture Probe (KAP).

  1. The Probe: Using ground-truth masks from datasets like COCO, they keep the pooling region (the object) fixed but vary the "aperture"—the amount of visible context around the object. They sweep from $d=0$ (only pixels inside the mask are visible) to the full image.
  2. The Metric: They summarize a model's context-resistance with a single scalar: $d_0/\text{full}$. This is the ratio of retrieval accuracy with zero context versus full context. A higher number means the model is more "context-resistant."
  3. Aperture Distillation: To fix the problem, they propose a training recipe. They use a "teacher" model that sees a wide aperture and a "student" model (a lightweight rank-8 LoRA adapter) that sees a tight aperture. By forcing the student's descriptor to match the teacher's, the model learns to focus its representation on the object itself rather than the background.
The KAP protocol: the pooling region is fixed at the object mask, while the visible context aperture expands.

Datasets

The study utilizes several key benchmarks:

  • COCO-Panoptic & ADE20K: Used for the primary KAP diagnostic and cross-dataset validation.
  • INSTRE: An instance-level retrieval dataset used to test the proposed fix under realistic conditions with predicted masks (from Segment Anything Model 3).
  • ILIAS: A large-scale instance retrieval benchmark used to demonstrate the generalizability of the findings.

Key Insights and Ideas

  • Object-Only is Not Context-Free: Across ten modern backbones, "object-only" descriptors retained only 28% to 67% of their full-image accuracy. Even the best models are heavily context-dependent.
  • The "Plain-Background" Crossover: When retrieval fails due to context removal, the descriptor doesn't just become random noise. Instead, it drifts toward categories that typically appear on plain backgrounds (like "kites" in the COCO dataset). The model essentially interprets the "blank" fill as a meaningful part of the object's identity.
  • Not Just a Transformer Issue: While one might blame Transformer self-attention, the authors found that convolutional networks (like ConvNeXt) also show substantial context reliance due to their large effective receptive fields.
  • Small Objects Suffer Most: Context reliance is strongly graded by size. Large objects can be retrieved fairly well without context (~80% retention), but small objects almost completely collapse (~20% retention).
  • A "Free Lunch" Fix: The LoRA adapter doesn't just help when images are masked; it actually improves full-image retrieval accuracy. By making the descriptor more robust to context removal, it forces the model to learn better, more intrinsic object features.

Key Results

  • DINOv2-G/14 emerged as the most context-resistant model tested, with a $d_0/\text{full}$ ratio of 67%.
  • MAE (Masked Autoencoder) models performed poorly, with very low absolute accuracy and high context dependence.
  • The LoRA adapter provided significant boosts: on the INSTRE dataset, it improved mean Average Precision (mAP) by +3.6 to +11.9 points across different backbones.
  • In transfer tests to the ILIAS dataset, the adapter nearly doubled the object-only retention for some models (e.g., EVA-02-MIM went from 36.3% to 70.9%).

Limitations and Future Work

  • Metric Artifacts: The $d_0/\text{full}$ ratio can be misleading for models with very low baseline accuracy (like MAE), as a low denominator inflates the ratio.
  • Architecture Scope: The study primarily focused on one CNN (ConvNeXt) and several ViTs. More work is needed to see if these findings hold across a wider variety of architectures.
  • Oracle vs. Predicted Masks: While the diagnostic uses "perfect" oracle masks, real-world systems use predicted ones. The authors showed their fix works with predicted masks, but the diagnostic itself still relies on ground truth for precision.
  • Transformer-Centric Fix: The current adapter is a LoRA variant designed for Transformers; a similar "fix" for CNNs remains for future work.

Summary updated

Abstract

Object-centric retrieval pipelines often mask the background before pooling backbone features, aiming for a descriptor that reflects the object rather than its scene. Whether this truly yields an object-only descriptor, however, is rarely tested. We introduce the Keyhole Aperture Probe (KAP), a counterfactual diagnostic that repurposes the ground-truth masks of existing panoptic segmentation datasets (COCO, ADE20K) for retrieval: it holds the pooled object region fixed and varies only the pixels outside it, summarizing a backbone's context dependence in a single number. We find that the pooled object region has already integrated its surround. Replacing the context, with a neutral fill or even a different real scene, changes the descriptor, and when retrieval fails under a blank fill, the descriptor matches plain-background images rather than its own object. Object-only is not context-free. Across ten modern backbones, object-only descriptors retain only about a quarter to two-thirds of their full-image accuracy, and the effect is not unique to Transformer self-attention: a convolutional backbone shows it too. The dependence is also reducible: a lightweight adapter that encourages the descriptor to stay stable as context is removed lowers it and improves full-image instance retrieval on INSTRE, transferring to ILIAS with accuracy improved or preserved. A matched contrastive-fine-tuning control does not reproduce those gains. We release KAP as a simple, backbone-agnostic evaluation protocol to measure and reduce this dependence before deployment.
BibTeX
@misc{meivar2026object,
  author = {Boaz Meivar and Shai Avidan},
  title = {{Object-Only Is Not Context-Free: A Counterfactual Diagnostic for Context Removal in Object Retrieval}},
  year = {2026},
  eprint = {2611.01962},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.01962},
}
Figure