Letting Other Images Reopen a Bad Crop: Cross-Image Part Evidence for Correcting Coarse Stages in Part Parsing

Many vision systems split hard labeling problems into stages. A coarse stage first finds the object, and a fine stage then labels its parts inside the region the coarse stage selected. The weakness is well documented: when the coarse stage clips a leg or a tail, the fine stage never sees it, and nothing downstream can bring it back. A separate line of work shows that labeling a group of related images jointly beats labeling each one alone, because clear images can resolve ambiguity in degraded ones. We propose to join these two observations at one specific point. Parts that are found confidently in other images of the same category are matched back into each image over its full extent, not only inside its coarse crop. When the image contains a strong match for an expected part outside its coarse mask, or a large region inside the mask that matches no part, that evidence is used to reopen and revise the coarse decision. The fine stage then runs again.

The claim is narrow and testable. Group consensus computed at the object level mainly fixes errors about what the object is. Group consensus computed at the part level fixes a different kind of error: truncation of thin, peripheral structures, where object-level consensus is known to be weak. If the claim holds, the gains should sit almost entirely on cascade-induced part errors. A coarse stage that is merely group-aware should not reproduce them.

What exists and what is missing

The relational side supplies the motivation and some of the machinery. Co-Salient Object Detection with Semantic-Level Consensus Extraction and Dispersion computes a category-level consensus from a CLIP vision transformer and dispatches it to each image in the group. It reports +1.9% max E-measure and +2.0% max F-measure on CoSOD3k. The authors also acknowledge that high-level consensus degrades fine boundaries on complex silhouettes. That admission is the motivating gap: object-level consensus is the wrong granularity for recovering a clipped tail.

Advancing Referring Expression Segmentation Beyond Single Image shows how large the payoff of group context can be: 75.32% versus 24.38% mIoU on G-RefCg. It also shows how easily consensus leaks false foreground into images that lack the target, which required a dedicated training scheme. That is a constraint our design must respect.

FlowWeb enforces cycle consistency over a fully connected graph of image correspondences, so that a match from image A to B to C must lead back to A. We reuse that idea as a cheap filter on part matches, applied to part prototypes rather than to dense pixel flow.

The cascade side supplies the failure mode. A Part-Based Deep Neural Network Cascade Model for Human Parsing works by scanning the whole image and then zooming into crops. It reaches 59.93% mIoU on ATR, a 4.58-point gain over FCN-8s. Its fine networks are blind to anything the scan clips. The same paper reports that adding a conditional random field (a pixel-smoothing post-process) hurt thin parts. This is a negative result that rules out "fix it with smoothing."

Three existing methods reduce cascade damage, all within a single image:

None of these lets other images testify against a coarse decision.

The work closest to ours connects the two directions without doing this. SCOPS learns parts across an image collection. However, it relies on an external saliency map, so a wrong foreground boundary cascades straight into the parts with no correction path. Unsupervised Hierarchical Semantic Segmentation with Multiview Cosegmentation and Clustering Transformers builds a part-to-object hierarchy, but its "co-segmentation" uses augmented views of one image. It also groups strictly bottom-up, with no feedback. The video methods (DeAOT, HCPN) propagate along time and cannot use unordered collections. Our contribution is the specific loop that none of these contain: cross-image part evidence revises the parent object decision.

Research Program

The system has three pieces, each built on top of an unchanged object-then-part cascade.

The part bank. For a group of $N$ images of one category, every image is first run through the cascade. From each image, features of pixels predicted with high confidence for part $k$ are pooled into a per-image part descriptor. These descriptors are combined, weighted by confidence, into a group prototype for each part. Comparing each image against $K$ prototypes keeps the cost linear in $N$. This avoids the quadratic cost of all-pairs pixel attention that limits FlowWeb-style graphs.

Match filtering. Prototype matches are kept only if they are mutually nearest, which is a two-image cycle check in the spirit of FlowWeb. Contributions from images whose own object confidence is low are down-weighted. This gives some protection against outliers and negative images in the group.

Revision evidence. For each image we compute two signals over its full extent:

  • Omission. Some of an expected part's prototype-match mass lies outside the coarse mask. A part counts as expected when most group images whose visible parts resemble this image's visible parts also show it, for example legs whenever head and torso are found.
  • Commission. Some area inside the mask matches no part prototype.

A small learned revision head takes these maps and the current mask and outputs a revised mask. The image is then re-cropped and the fine stage re-run, for at most two iterations. Following stacking, the revision head is trained on held-out cascade outputs, so it sees realistic coarse errors rather than near-perfect training-set masks.

Error analysis. The analysis component matters as much as the module. We will decompose every part error into three categories:

  • cascade-induced: the ground-truth part pixel lies outside the coarse mask;
  • in-mask;
  • category confusion.

We will also measure the oracle ceiling obtained by substituting the ground-truth object mask. This tells us, before any method comparison, how much part error the coarse stage is responsible for. That number bounds what any coarse-stage correction can achieve.

Baselines

The naive reference point is a single-image, single-pass part segmenter on the same backbone with standard augmentation, with no cascade and no group. Next comes the plain cascade with no feedback, which shows how much cascading costs relative to the single pass.

The strongest existing single-image competitor is the two-stream joint object-part model of Joint Object and Part Segmentation Using Deep Learned Potentials. It removes the cascade instead of repairing it. That paper reports roughly 13% over SPS, 4.5% over Hypercolumns on Horse-Cow, and 6.2% over FCN-16s on PASCAL Quadrupeds. All of these are relative gains; no reported absolute part mIoU was found on Lacuna for Horse-Cow or PASCAL Quadrupeds. We will therefore re-implement it on our backbone rather than quote it. Gated feedback refinement (G-FRNet) is run the same way. Its Horse-Cow result is reported only qualitatively in the file.

The strongest relational competitor is a cascade whose coarse stage is replaced by the group-aware co-salient detector of Xu and Mu. This baseline uses the same groups and the same fine stage.

Two ablations isolate the key ingredient:

  • Self-prototypes only. The revision head runs with the image's own part prototypes. This keeps the feedback loop but removes the group.
  • Object-level consensus only. The revision is driven by object-level consensus alone. This keeps the group but removes part granularity.

All methods share the backbone, training data, group composition, and a fixed hyperparameter search budget. Inference compute is reported, since our loop re-runs the fine stage. "Beating" a baseline means a higher part mIoU averaged over three seeds that is concentrated in the cascade-induced error bucket, at no loss on the no-target negative images.

Why not just…?

Why not just make the coarse stage group-aware and leave the cascade alone? This is the most serious threat. If most cascade errors are category confusions, a co-salient coarse stage will remove them, and part-level feedback adds nothing. Our prediction differs because Xu and Mu report boundary degradation from semantic consensus, and truncation of thin parts is exactly a boundary failure. The error decomposition settles the question. If the co-salient coarse stage closes the cascade-induced bucket as well as our loop does, we drop the proposal's main claim.

Why not just remove the cascade with a joint single-image object-part model? Joint potentials do let parts correct objects. But an occluded or blurred leg gives a single image no part evidence to correct with, and that is where other images help. We expect the joint model to match us on clean images and fall behind on images flagged as degraded. If it does not fall behind, the group is unnecessary.

Why not just dilate the crop or smooth the output? Dilating the crop trades truncation for clutter. Smoothing is ruled out by the finding in the human parsing cascade paper that conditional random fields erased sunglasses and belts. We include a tuned crop-margin baseline. It is cheap and may recover part of the gain honestly.

Why not just use augmented views of the same image, as in hierarchical segment grouping? Augmented views share the same occlusion, so they cannot supply a leg that is hidden. The self-prototype ablation with test-time augmentation measures exactly this.

Proposed Experiments

Supervised part parsing. The main testbeds are Horse-Cow Parsing, PASCAL Quadrupeds, and PASCAL-Part, with groups of $N$ images sampled per category. As a secondary check outside animals, we use ATR human parsing, where the cascade baseline reaches 59.93% mIoU.

Contaminated groups. To test leakage, we inject a fraction of certified-negative images into each group, in the style of the GRD benchmark. We report false-positive area on the negatives.

Object-level check. CoSOD3k checks whether revised object masks also improve co-saliency metrics.

Metrics. We report overall part mIoU, per-part IoU for thin peripheral parts, the fraction of the oracle gap closed, and inference time.

Decisive checks. Four outcomes would falsify the idea:

  • groups of shuffled, mixed categories give the same gains;
  • gains do not grow from $N=1$ to $N\approx 16$;
  • improvement is spread evenly across error buckets instead of concentrated in cascade-induced errors;
  • the co-salient coarse-stage baseline matches us on truncation errors.