Vision models that do well on curated images often fail in the field. A plant-disease classifier trained on leaves photographed against plain lab backgrounds stops working on photos taken among soil, shadows and overlapping foliage. A tiger re-identification model breaks when the camera changes. A lung-ultrasound scorer learns to read the patient's fat thickness instead of the lung. The standard fix is to segment the region of interest first and hide everything else. That fix needs a segmenter: either trained on expert masks, or built by hand from edge filters. Both fail on exactly the hard field images where they are most needed.
We propose to remove that dependency. The model breaks the whole scene, not just the object, into many discovered parts, and then learns which of those parts make up the region of interest. A part is kept when the task prediction built from it works equally well across acquisition environments such as cameras, sites, or lab versus field. Background clutter is never forced into a single "background" slot. It gets its own parts, so it can be recognized and switched off rather than leaking into the object parts.
Core Claim
Unsupervised part discovery methods break down in the field because they assume a clean figure-ground split before discovery begins. Unsupervised Part Discovery from Contrastive Reconstruction learns parts that match across instances and transform consistently under image warps. It explicitly depends on a foreground saliency map and reports that noisy separation degrades its parts. Unsupervised Part Discovery via Dual Representation Alignment (PartFormer) replaces the transformer's class token with $K$ part tokens plus one background token, and is evaluated only on curated, centered-object datasets. Real field backgrounds are structured: soil, mulch, neighboring leaves, the fat layer above the lung. One background token cannot absorb all of that, so the leftovers drift into the object parts. This is the "attention drift" that Adaptive Attention Distillation for Robust Few-Shot Segmentation under Environmental Perturbations documents for attention maps under blur, camouflage and clutter.
Our claim has two parts. First, giving the model an overcomplete set of scene-level part tokens, with no saliency mask, lets clutter separate into its own parts. Second, a learned on/off gate per part, trained so that the classifier's error is equal across training environments, selects the parts that form the region of interest. Together, these produce region-of-interest masks that reduce the lab-to-field gap as well as or better than supervised teacher masks, using only image-level labels and environment identifiers. A side effect is that overcomplete tokens plus gating soften the fixed-$K$ problem both part-discovery papers share.
Research Program
The model is a Vision Transformer (an image model that treats patches as tokens and mixes them with attention) with $M$ learnable part tokens, where $M$ is set larger than the expected number of object parts (16 to 24 by default). Discovery losses are reused from PartFormer and Contrastive Reconstruction: swapping part representations across two augmented views to reconstruct the image, concentration and area losses that keep each part spatially compact, and equivariance under geometric warps. We drop the foreground mask and the dedicated background token.
To these losses we add photometric field perturbations to the paired views: blur, glare, shadow casting, sensor noise, and pasting the scene into new clutter. Part assignments must stay stable under these perturbations. This addition alone is a cheap baseline, and we test it as one.
Each part $k$ receives a scalar gate $g_k \in [0,1]$, computed from that part's pooled feature. The downstream descriptor is mask-weighted pooling, in the spirit of Computed tomography–based COVID–19 triage through a deep neural network using mask–weighted global average pooling. The difference is that the weighting mask is the gate-weighted sum of discovered part maps rather than a supervised organ mask. The gates are trained with
$$\mathcal{L} = \frac{1}{|E|}\sum_{e \in E} R_e + \lambda \,\mathrm{Var}_{e \in E}\left[R_e\right] + \mu \sum_{k=1}^{M} g_k + \mathcal{L}_{\text{part}},$$
where $E$ is the set of training environments, $R_e$ is the task loss on environment $e$ computed from the gated descriptor, the variance term penalizes features whose usefulness differs across environments, the third term pushes toward few active parts, and $\mathcal{L}_{\text{part}}$ is the discovery objective above. The intuition is as follows. A background part like plain lab backdrop or a camera-specific scene element helps in one environment and not another, so the variance term turns it off. A part on the leaf, the tiger's flank stripes, or the pleural line helps everywhere, so it stays on.
When no environment labels exist, only the task and sparsity terms act. We report that setting separately because it is the honest limit of the method.
We then analyze three questions. Do the gated regions match ground-truth object or lesion masks? Do the surviving parts stay stable under perturbation? Which parts get switched off, and do those correspond to known shortcuts?
Baselines
The naive reference is a ViT-Base trained end-to-end on the task with standard augmentation (random crops, color jitter, CutMix), plus the same photometric field perturbations we use, so that our perturbations are not an unearned advantage. It uses the same initialization (self-supervised DINO weights), the same epochs, and the same hyperparameter search budget as our model.
The strongest existing competitor for re-identification is PAW-ViT. It uses part tokens supervised by pseudo masks from a teacher ensemble of segmentation and pose models, and is the closest system to ours because it already combines part tokens, region supervision and wild imagery. On ATRW it reports a 3.2% cross-camera mAP gain and a 3.8% single-camera gain over ViT-Base, about 1% cross-camera over the prior best method, and a 4.6% mAP gain over PCN-RERP on the YakReID-103 hard split. The file gives these only as gains, so no absolute mAP is on record. We rerun PAW-ViT with its own teacher, on the same splits, at the same backbone size and with matched training budget. Beating it means matching or exceeding its cross-camera ATRW mAP and its YakReID-103 hard-split mAP without any teacher. A tie would already be a meaningful result, since the teacher is the costly and brittle component.
For part-discovery quality the reference is Contrastive Reconstruction, which reports NMI 46.0 on CUB-200-2011 versus SCOPS at 39.1. PartFormer reports large ARI gains over DINO clustering on CUB, but the file gives no absolute value. Both are run in their native mode with a saliency mask, and again with the mask removed.
Three ablations isolate our ingredients. The first keeps one background token and a saliency mask, which tests overcomplete scene parts. The second sets $\lambda = 0$, which tests the environment-invariance term. The third keeps our gating objective exactly but replaces learned parts with k-means clusters of frozen DINO patch features, which tests whether learning parts matters at all.
Why not just…?
Why not just crop with a supervised detector or a pretrained segmenter, then classify? This is the most serious alternative. PAW-ViT itself shows the cost: its authors report that teacher masks fail on uncommon wild poses and pass those errors into the part tokens. PlantSeg shows supervised segmenters reaching 46.24% mIoU with ConvNeXt-L and 10.22% with MobileNetV3 on in-the-wild lesions. Neither result rules out a strong general segmenter producing good outer object masks, though. We therefore include a detector-crop baseline trained on available box annotations, PAW-ViT's teacher masks, and a lesion segmenter trained on PlantSeg as an upper reference that uses mask supervision we do not. If teacher crops match us on held-out environments, the contribution shrinks to annotation savings, and we will say so.
Why not just cluster DINO features and pick clusters with the classifier? This could plausibly work as well, which is why the frozen-cluster ablation uses our exact gating objective. PartFormer reports that vanilla ViT features for different sub-parts are nearly indistinguishable and that it beats DINO clustering by a large ARI margin on CUB. That suggests clusters will merge object and clutter in the field. If frozen clusters with environment gating close most of the gap anyway, the right paper is about the gating criterion, not the discovery model, and we would reframe it that way.
Why not just rely on a plain ViT with heavy augmentation? The lung-ultrasound study The Role of Pleura and Adipose in Lung Ultrasound AI shows that unmasked end-to-end training reached AUC 0.6553 and attended to fat thickness, while hand-built pleural plus Merlin's-space masking reached 0.7742. Augmentation does not remove a shortcut that is present in every training image. Our augmented ViT-Base baseline measures this directly on every benchmark.
Why not just add perturbation consistency to PartFormer and keep its saliency mask? This is the incremental version of our idea. It is the first ablation, run on saliency maps computed from field images rather than oracle masks, so a saliency failure counts against it the same way it would in deployment.
Proposed Experiments
The first experiment is part discovery under hazards. On CUB-200-2011 and PartImageNet we measure NMI and ARI against keypoint and part labels in three conditions: clean, with saliency masks removed, and under a perturbed test set built from blur, glare, shadow and background-swap transforms. We also measure gated-mass precision, the fraction of gate-weighted area that falls inside the ground-truth object mask. The decisive check is whether our NMI degrades less than Contrastive Reconstruction's 46.0 clean figure does once its mask is removed.
The second is cross-camera re-identification. On ATRW, cameras serve as environments; on YakReID-103, capture sessions do. We measure mAP and rank-1 on the cross-camera and hard splits against PAW-ViT and ViT-Base.
The third is lab-to-field plant disease classification. We train with PlantVillage (lab) and PlantSeg training images (field) as two environments, on the disease classes that overlap after label mapping, and test on held-out PlantDoc images. Metrics are held-out accuracy and the lab-minus-field gap. On PlantSeg test images we check whether gated regions cover the annotated lesions, measured as lesion recall at fixed gated area, compared with class activation maps from the baseline.
The fourth is the lung-ultrasound rediscovery test, if access to the 210-video clinical cohort can be obtained. The question is whether, without the edge-filter pipeline, the gates switch off the subcutaneous layer and recover something close to the pleural plus Merlin's-space region and its AUC of 0.7742. Environments there are acquisition sessions if recorded; otherwise this is the no-environment setting.
Across all tasks, a background-swap test measures shortcut reliance: we paste the gated foreground onto backgrounds from other environments and count how often predictions flip.
The idea fails if any of the following holds: the frozen-cluster ablation matches the full model; setting $\lambda = 0$ changes nothing on held-out environments; or teacher-cropped baselines beat us on cross-camera ATRW.
References
- Unsupervised Part Discovery from Contrastive Reconstruction
- Unsupervised Part Discovery via Dual Representation Alignment
- Adaptive Attention Distillation for Robust Few-Shot Segmentation under Environmental Perturbations
- Computed tomography–based COVID–19 triage through a deep neural network using mask–weighted global average pooling
- An improved architecture for part-based animal re-identification through semantic segmentation distillation
- A Large-Scale In-the-wild Dataset for Plant Disease Segmentation
- The Role of Pleura and Adipose in Lung Ultrasound AI