Closing the Detector's Recall Gap in Crowded Scenes with Coverage-Driven Re-Prompting

Most systems that cut individual objects out of an image work in two steps. A detector first draws a box around each object it finds, and a mask model then traces the object's outline inside each box. The weakness of this design is simple: an object the detector misses can never be segmented, however good the mask model is. This matters most in crowded images such as retail shelves, cell fields, flocks of animals, and dense charts. There, boxes overlap so heavily that the detector's duplicate-removal step (non-maximum suppression, which deletes boxes that overlap a higher-scoring one) throws away real objects. We propose to close this gap with a feedback loop. Earlier refinement loops fed the current masks back into the network only to sharpen their shapes. Ours feeds them back to ask a different question: which object-like parts of the image does no mask explain yet? Those unexplained regions become new point prompts for a promptable segmenter, a model like Segment Anything that returns a mask for a clicked location. A small gate then decides whether each new mask is a new object, a fragment of an existing one, or a duplicate.

The bet is that the loop can change the set of instances, not just their outlines, and that this is where the gains sit in crowded scenes. If it works, recall will rise above the detector's own ceiling, and the improvement over standard pipelines should grow as the number of objects per image grows. We treat object density as the main evaluation axis rather than as a nuisance variable.

Core Claim

The prior work frames the problem precisely. Iterative Instance Segmentation feeds the previous mask heatmap back into the network as extra input channels. The network learns shape priors through this loop, filling in occluded parts and suppressing pixels of neighbouring instances. The same paper also shows that final recall stays strictly bounded by its Fast R-CNN detector. Later work hits the same ceiling: Segment Anything, Even Occluded cannot recover the full shape of an occluded object the upstream detector never boxed, and Top-Down Beats Bottom-Up in 3D Instance Segmentation reports the same pattern for 3D proposals. Bottom-up Instance Segmentation using Deep Higher-Order CRFs adds a second failure, truncation: a box that clips an object's limb also clips its mask. These papers motivate the proposal. They show that the loop and the detector have been kept separate, with refinement always running downstream of a fixed instance set.

Our claim is that a refinement loop can be built on coverage instead. At each round we compute a residual map, defined as an objectness score from a frozen self-supervised backbone multiplied by one minus the union of current masks. Peaks in this map are re-submitted as point prompts. Masks may also grow past their original boxes, which addresses truncation. The remaining design problem is deciding when two masks are the same object, and this is where semantic gating comes in.

The Gate

Two papers supply the method we build on. Hengcan Shi et al. check that two boxes share a semantic identity, using CLIP (an image–text embedding model), before suppressing one of them. No time to train! rescores candidate masks with "semantic-aware soft merging". This combines intersection-over-self (the fraction of one mask lying inside another) with cosine similarity between DINOv2 mask embeddings and class prototypes.

The same paper also reports a constraint. Semantic matching struggles to separate visually identical instances, such as adjacent sheep or identical packages, and those are exactly the cases that fill crowded scenes. We therefore do not let semantics decide alone. For a new candidate mask $m$ and an existing mask $m'$, the gate outputs

$$g(m, m') = \sigma\big(w^\top [\, s_{\text{sem}},\ \text{IoS},\ b,\ \rho,\ \rho\cdot s_{\text{sem}} \,]\big).$$

Here $\sigma$ is the logistic function and $s_{\text{sem}}$ is the cosine similarity of the two masks' DINOv2 embeddings. $\text{IoS}$ is the intersection-over-self of $m$ with $m'$, and $b$ is the mean image-edge strength along their shared border. $\rho$ is a label-free local density estimate: the number of residual peaks plus mask centroids within a fixed radius. $w$ is a small learned weight vector. A high $g$ means merge or discard $m$, and a low $g$ means keep $m$ as a new instance.

The interaction term $\rho\cdot s_{\text{sem}}$ is the key ingredient. It lets the gate learn to trust semantic similarity less when the neighbourhood is crowded. In those neighbourhoods "same category" stops implying "same object", and a visible boundary $b$ should outweigh it. This is close in spirit to the instance-dominance gating of IFAdapter, which assigns features in overlap zones during image generation. Here the gate operates on mask identity during recognition instead. The gate has five weights, so it can be fit on a few hundred labelled mask pairs. We also report a training-free version with hand-set weights so the contribution of learning is visible.

Research Program

The system has four parts. The first is an off-the-shelf detector with a box-prompted promptable segmenter, as in SAMEO-style pipelines. The second is the residual computation on frozen DINOv2 features. The third is the gate. The fourth is a stopping rule: halt when the residual's peak mass falls below a threshold or after a fixed number of rounds.

Two analyses accompany the main results. First, we measure the "ceiling breach": the fraction of correctly segmented ground-truth objects that no detector box covered at IoU 0.5. This directly tests whether the loop escapes the detector bound documented by Ke Li et al.. Second, every metric is reported binned by ground-truth instances per image. The visual-complexity literature shows that failures under density are perceptual rather than downstream. For example, ChartMuseum reports human accuracy of 93.0% against 63.0% for the best model, and only a 1.4% gain from extended reasoning. A segmentation fix should therefore show its value as a flattening of the curve against density, not as a single averaged number.

Evaluation uses four settings. COCO instance segmentation serves as the general reference. A dense retail-shelf collection with near-identical products tests the fine-grained confusion case. A dense cell-microscopy set tests crowding without category cues. Chart-element segmentation tests the element-parsing regime where ChartLens notes that parsing is bounded by box proposals.

Baselines

The naive reference is the detector plus box-prompted segmenter run once at standard thresholds. This is the pipeline whose ceiling we claim to break. Its recall on the dense bins sets the bar any loop must clear.

The strongest existing method is the semantic-aware soft merging of No time to train!. It is the closest published combination of a promptable segmenter with DINOv2-based semantic arbitration of overlapping masks. We will run its released procedure with the same backbone and the same segmenter. We give it the same detector, lowered to the same low confidence threshold our method uses, so that both systems start from an identical candidate pool. The file contains no number for this method on our dense benchmarks or on COCO: no reported result found on Lacuna for No time to train! on COCO or dense-scene instance segmentation. We will therefore reproduce it. Beating it means higher average recall at 100 masks (AR@100) and mask average precision (AP) in the top two density bins, at matched segmenter calls, with no loss in the sparse bins.

We also run the CLIP dual-constraint suppression of Hengcan Shi et al. in place of non-maximum suppression. This is the most direct fix to the box-suppression failure. Again, no reported result found on Lacuna for it on our benchmarks.

The shape-feedback loop of Iterative Instance Segmentation is re-implemented on the same segmenter as a mask-feedback baseline. It refines shapes but cannot add instances.

Three ablations isolate our ingredients. The first removes the residual re-prompting, keeping the gate on detector outputs only. The second replaces the gate with plain IoU merging. The third drops the density terms $\rho$ and $\rho\cdot s_{\text{sem}}$. All methods share data, backbone, a fixed segmenter-call budget, and the same small labelled pool for any tuning.

Why not just…?

Why not just lower the detector threshold and use soft suppression? This is a real threat. Many missed objects are scored but suppressed rather than never seen. Our design absorbs it: every competitor starts from the same lowered-threshold pool. The ceiling-breach metric then counts only objects absent from that pool. If ceiling breach is near zero and gains vanish under matched thresholds, the residual loop adds nothing, and we will report that.

Why not just prompt the segmenter on a dense grid and merge, skipping the detector entirely? Grid prompting is detector-free, so it avoids the recall ceiling outright. Its costs are many fragments and duplicates, plus a fixed resolution that may still skip small packed items. It is the most dangerous alternative because, with a good merge step, it could match us. We therefore compare at equal numbers of segmenter calls, giving the grid our gate as its merge step. The claim we test is that residual-targeted prompts reach the same recall with far fewer calls, and higher recall at equal calls in dense bins. If the grid wins at matched calls, the contribution reduces to the density-aware gate alone, and the note's framing would change accordingly.

Why not just fine-tune the detector on crowded data? Fine-tuning needs dense box labels for each domain, while our gate needs a few hundred mask-pair labels and the rest is frozen. It also leaves the hard box crop that causes truncation, the failure noted by Arnab and Torr. Where dense labels exist, as on the retail set, we include a fine-tuned detector as an upper reference. We also run our loop on top of it to check whether the gains are complementary.

Decisive Checks

The proposal succeeds if three things hold. First, the ceiling-breach fraction is clearly positive on the dense retail and cell sets. Second, the AR@100 gap over the reproduced No-time-to-train baseline widens monotonically with instance count. Third, removing the density interaction term specifically hurts the near-identical retail bins while leaving COCO's sparse bins unchanged. If the gap stays flat across density, or grid prompting matches us at equal calls, the coverage-driven loop is not the source of the gain. We would then report the gate as a standalone merge improvement.