Self-supervised vision models learn without labels by making two altered copies ("views") of an image produce matching representations. This is contrastive learning: views of the same image are pulled together and views of different images are pushed apart. Multi-scale variants add small local views alongside the full image so the model also keeps fine spatial detail. Today those local views are chosen blindly, as random crops, a fixed 3×3 grid, or region proposals from an off-the-shelf heuristic. On cluttered scene photos this often pairs a dog with a chair, or an object with empty sky. We propose to let the model's own loss decide where local views go and how large they should be. We measure which image regions, at which level of blur, actually drive agreement between views, and we draw the next local crops there at the matching size.
The measuring tool comes from gradient-based attribution, the family of methods that explain a network's output by differentiating it with respect to its input. One member, Blur Integrated Gradients, integrates gradients along a path that starts from a fully blurred image and gradually sharpens it. This yields an importance map over both location and spatial scale. We apply it, cheaply and periodically, to the contrastive agreement score rather than to a class logit. Its per-pixel "characteristic scale" then sets local crop sizes. The claim to test is this: on multi-object scene data, local views placed and sized by scale-space attribution of the self-supervised loss transfer better to detection and segmentation than random, grid, or activation-clustering views at equal compute. The key ingredient is the scale axis, not just the location.
Core Claim
Existing work splits along two lines. Attribution in Scale and Space (BlurIG) shows that integrating gradients along a Gaussian blur path avoids the "ghost" artifacts of black-image baselines. It also shows that classifiers recognize coarse concepts at large blur scales and fine ones, such as dog breeds, only at fine scales. It is purely a post-hoc diagnostic for supervised classifiers. Separately, Boosting Graph Contrastive Learning via Graph Contrastive Saliency (GCS) takes the gradient of a contrastive objective to find salient subgraphs, then builds positives that keep them and hard negatives that destroy them. It lifts unsupervised TU-benchmark accuracy to 77.39% from GraphCL's 75.71%. GCS works on discrete graph structure and has no notion of spatial scale.
The vision methods closest to us choose regions without gradients. Saliency Guided Contrastive Learning on Scene Images (SGCL) runs a normalized-cut segmentation on a single high-level feature map. UniVIP relies on offline Selective Search proposals. Neither tells the crop sampler at what size a region carries the evidence the loss uses. Our proposal differs from GCS in domain and in the scale axis, and from SGCL and UniVIP in using loss sensitivity rather than activation similarity. Unlike everything in the file, we use the attribution's scale coordinate as a sampling decision.
The mechanism is simple to state. Let $x_\sigma = G_\sigma * x$ be the image blurred by a Gaussian of width $\sigma$. Let $s(\cdot)$ be the agreement score between an online-network embedding of a view and the target-network embedding of a second view of the same image (cosine similarity for a BYOL-style learner). Because the Gaussian obeys the heat equation, $\partial x_\sigma / \partial \sigma = \sigma \, \Delta x_\sigma$, where $\Delta$ is the spatial Laplacian. The scale-space attribution at pixel $p$ and scale $\sigma$ is therefore
$$A(p,\sigma) = \big[\nabla_{x_\sigma} s(x_\sigma)\big]_p \cdot \sigma\,\big[\Delta x_\sigma\big]_p ,$$
and BlurIG's attribution is the integral of $-A(p,\sigma)$ from the maximum blur down to $\sigma = 0$. For each location we record the total attribution, which sets where to crop, and the scale $\sigma^*(p)$ at which most of that attribution accumulates, which sets crop size. Following the standard scale-space link between blob width and Gaussian scale, a local crop centered at $p$ gets side length proportional to $\sigma^*(p)$. The path view matters here for a specific reason. Late in contrastive training, positive pairs already agree almost perfectly, so the plain gradient saturates toward zero. Integrating from the blurred image avoids this, as the IG literature argues.
Research Program
The work proceeds in three stages, ordered so the idea can fail early and cheaply.
The first stage is an audit. We take publicly reproducible BYOL and DetCo-style ResNet-50 models pre-trained on COCO and compute $A(p,\sigma)$ for the contrastive agreement score on held-out COCO images. We ask whether the characteristic scale $\sigma^*$ varies meaningfully across images and across objects within an image. If it is nearly constant, a fixed crop-size schedule would capture everything and the proposal reduces to location-only saliency; in that case we stop. The same audit addresses an open question raised by DetCo: does intermediate-layer supervision change where and at what scale the loss draws its evidence? DetCo found that such supervision helps detection but costs final-layer linear accuracy. We compare attribution maps taken at its intermediate stages with those from a single-level model. To check that the maps mean something before we train on them, we compare them against COCO instance masks and run deletion tests. In a deletion test, the top-attributed pixels are blurred out and we measure how fast agreement drops.
The second stage builds the training method on a BYOL learner over COCO. The attribution computation is amortized. Every $K$ epochs, the momentum (target) network computes an 8-step blur-path attribution for each image and caches a coarse map of location-by-scale importance. Local views are then sampled from this cache. Most local crops are drawn with probability proportional to attribution mass and sized by $\sigma^*$, while a fixed fraction remain uniformly random, so the model cannot lock onto its own early mistakes. Taking a GCS-style step, we also mask the top region and re-attribute once to recover secondary objects. This guards against the saliency bias toward dominant objects documented by Attribution Guided Factorization. Local views are contrasted against the global view as in DetCo's global-to-local term.
The third stage measures cost honestly. SSMix reports roughly a 2× slowdown from a single extra saliency pass per step. With 8 blur steps every $K$ epochs, the amortized overhead is about $8/K$ extra forward-backward passes per image-epoch. We report wall-clock and GPU-hours and choose $K$ so the overhead stays near the roughly 35% that UniVIP already pays for Selective Search and optimal transport.
Baselines
The naive reference point is BYOL with standard random multi-crop local views, pre-trained on COCO. SGCL reports this baseline as the one it improves by +0.7 AP on COCO detection and +0.4 AP on instance segmentation. Every method in the comparison starts from this exact code path. Beating it is necessary but tells us little on its own.
The strongest competitor for location selection is SGCL (Saliency Guided Contrastive Learning on Scene Images). It asks the same question we do, which crops to trust on cluttered scenes, and answers it with normalized cuts on forward features. SGCL reports 61.8% ImageNet linear top-1 after COCO+ pre-training and 36.1% top-1 with 1% of ImageNet labels, which it describes as +4.3 over ORL. We re-implement SGCL inside our BYOL code with the same backbone, augmentations, epochs, and a matched hyperparameter search of the same size. This removes code differences. Beating SGCL means a gain in COCO box and mask AP and in the 1%-label probe that exceeds seed-to-seed variation over three seeds.
The strongest competitor on absolute detection numbers is UniVIP (UniVIP). It reports 40.8 mAP on COCO detection after COCO pre-training, against 38.9 for ImageNet-supervised. Running UniVIP's full optimal-transport pipeline is expensive. We therefore treat its reported number as an external reference and also run its Selective Search proposals as a drop-in crop source in our sampler, which gives a clean comparison of region sources. DetCo's fixed 3×3 grid serves as the multi-scale-by-construction baseline. DetCo reports 58.2 VOC mAP with ImageNet pre-training, which is not directly comparable to our COCO setting, so we run the grid inside our pipeline.
Two ablations isolate the key ingredient. The first keeps attribution-chosen locations but draws crop sizes from the baseline's random size distribution. This removes the scale axis. The second replaces the blur-path integral with a single plain gradient at the unblurred image, as GCS and SSMix do. This removes the path and tests the saturation argument. All runs are matched on total GPU-hours. Baselines that do not pay attribution overhead get extra epochs to use up the difference, since DetCo and MimCo both show that epochs strongly move transfer results.
Why not just…?
Why not just use random multi-scale crops or DetCo's grid? On iconic, single-object images this may well be enough. UniVIP and SGCL both argue that on non-iconic scenes random crops break the assumption that two views share content. SGCL's gains over BYOL support that argument. Our naive and grid baselines on COCO test it directly. If neither attribution variant beats them at matched compute, the proposal fails.
Why not just use self-attention maps or normalized cuts, which are nearly free? This is the most serious threat for location selection. The file's evidence against attention, from Dynamic Prompt Corruption, comes from language-model reasoning, not vision self-supervision, so we do not lean on it. We expect activation-based maps to find objects about as well as gradients do. We concede that ground. The proposal therefore rests on the scale coordinate, which neither attention nor normalized cuts provides. The decisive comparison combines SGCL's location map with our attribution-derived sizes and sets that against SGCL's location map with random sizes. If attribution-derived size adds nothing, we will report that the gradient machinery is unnecessary and that cheap activation saliency suffices.
Why not just use a single gradient instead of a blur path? It costs one pass instead of eight and is what GCS and SSMix use. It saturates, though, once positives agree strongly, and it gives no scale coordinate unless one differentiates through a blur anyway. The plain-gradient ablation measures how much the path buys. If the gain is small, we keep the cheaper variant with a two-step blur probe.
Why not just train the baseline longer? Extra epochs are a real alternative use of the same compute. That is why every comparison is matched on GPU-hours rather than epochs.
Proposed Experiments
All models are ResNet-50 BYOL variants pre-trained on COCO train2017 for an equal compute budget. Transfer is evaluated with Mask R-CNN on COCO (box and mask AP, 1× schedule), Faster R-CNN on PASCAL VOC (mAP), ImageNet-1K linear probe top-1, and the 1%-label ImageNet probe on which SGCL reports its largest gain. Attribution quality is checked separately. We report overlap between attribution mass and COCO instance masks, deletion-curve area, and how often the masked re-attribution recovers secondary objects. These are measures that Attribution Guided Factorization motivates but for which the file gives no numeric AUC.
Three outcomes decide the proposal. The first is the audit: the characteristic scale must vary across images and objects in pre-trained COCO models, or we stop. The second is the size ablation: attribution-derived crop sizes must beat random sizes with locations held fixed, from either our maps or SGCL's. This is the test of what is new here. The third is the matched-compute comparison: the full method must beat our SGCL re-implementation on COCO AP and the 1%-label probe across three seeds. If the second check fails while the third passes, the result is a GCS-style location sampler for images. That is still useful, but it is a narrower claim than proposed, and we would report it as such.
References
- Attribution in Scale and Space
- Boosting Graph Contrastive Learning via Graph Contrastive Saliency
- Saliency Guided Contrastive Learning on Scene Images
- UniVIP: A Unified Framework for Self-Supervised Visual Pre-training
- DetCo: Unsupervised Contrastive Learning for Object Detection
- Visualization of Supervised and Self-Supervised Neural Networks via Attribution Guided Factorization
- SSMix: Saliency-Based Span Mixup for Text Classification
- Improving Complex Reasoning with Dynamic Prompt Corruption: A Soft Prompt Optimization Approach