Choosing SAM Masks by Class Prototypes in Point-Supervised Oriented Detection

Training a detector for aerial images normally requires a human to draw a rotated box around every ship, plane and bridge. That is slow, so recent methods ask only for a single click per object, labeled with its class. They turn each click into a rotated box by prompting the Segment Anything Model (SAM, a class-agnostic segmenter that returns several candidate masks for a prompt point). The weak link is choosing which candidate mask to trust. Current pipelines pick it with hand-written geometric rules such as rectangularity, color consistency and distance to the click. These rules fail on irregular or isolated objects like bridges, harbors and track fields. They also never use the one piece of semantic information the annotator already supplied, which is the class of each click.

We propose to spend that information. Every labeled click gives a feature vector from a frozen pre-trained encoder (DINO-style self-supervised features) at a location known to belong to a given class. Pooling these across the whole training set yields several prototypes per class, meaning reference feature vectors that summarize what that class looks like. We also build a set of background prototypes. Each SAM candidate is then scored by how much of the mask is classified as the clicked object's class (purity) and how much of the surrounding same-class region it covers (coverage). The best-scoring mask becomes the pseudo-box that trains the detector. In two sentences, the idea is to replace geometric guesswork in mask selection with a nearest-prototype vote, and to build those prototypes for free from the class labels attached to the clicks. The claim is that this produces more accurate rotated pseudo-boxes, most visibly on the irregular classes where heuristics break.

Core Claim and Position Relative to Prior Work

The proposal sits on a narrow but real gap. P2RBox prompts frozen SAM with points, picks masks with a per-image multiple-instance learning module, and gets the angle from the principal eigenvector of the mask. Point2RBox-v3 routes sparse scenes to SAM and dense ones to watershed, and picks among SAM's outputs with hand-crafted priors. Semantic-aware SAM (SAPNet) names the two failure modes we target. The "local issue" is choosing a part instead of the whole object, and the "group issue" is merging neighbors. SAPNet fixes them with point-distance guidance and box mining on natural images. None of these three keeps a class-level feature memory, so mask selection never asks whether a mask looks like the class that was clicked.

On the prototype side, the closest work is ZODS-RS. It purifies prototypes from frozen DINOv3 features with robust scatter estimation and matches them to SAM2 proposals. However, it is training-free, uses a single reference image per class, and outputs only horizontal boxes. Its authors list rotated-box output and adaptation to weak labels as open problems. We use ZODS-RS's robust prototype estimation as a supplied method. We change the regime: our prototypes come from thousands of class-labeled clicks rather than one reference, and they act as a pseudo-label selector inside a trained oriented detector rather than as the detector itself.

Weakly Supervised Semantic Segmentation by Pixel-to-Prototype Contrast and UNICL-SAM motivate the multi-prototype design. The first shows pixel-to-prototype comparison expands coverage beyond discriminative hot-spots. The second shows that part-level prototypes beat a single pooled one on frozen SAM features. We take the lesson without their requirements: we need neither image-level activation maps nor ground-truth support masks.

Two results constrain the design. First, Direction 2's known domain-shift problem means frozen natural-image features may separate aerial classes poorly, so we test separability before anything else. Second, Point2RBox-v2 reaches 86.9% on ships against 87.1% with full rotated boxes. Dense, regular classes therefore have little headroom, so our gains must come from irregular and isolated classes.

Research Program

Prototype construction runs once, offline. A frozen encoder produces dense patch features over each training image; DINOv3 is the default, and SAM's own encoder is an ablation. We then take the feature at each clicked point, averaged over a small window to tolerate off-center clicks. For each class we cluster these features into $M$ prototypes with k-means and apply the robust re-estimation of ZODS-RS to remove mislabeled or off-object clicks. Using several prototypes captures multimodal appearance, such as planes viewed at different angles or docked versus open-water ships, which a single centroid in the style of Prototypical Networks would average into a low-density point. Background prototypes come from clustering features sampled far from every click. We discard any background cluster that sits close to an object prototype, since it may contain unclicked objects.

Every pixel $p$ is then assigned to its nearest prototype $a(p)$ across all classes and background. For a click of class $y$ and a SAM candidate mask $m$, let $R_y$ be the connected region of pixels assigned to class $y$ that contains the click. The selection score is

$$s(m) = \frac{2\,\pi(m)\,\kappa(m)}{\pi(m)+\kappa(m)}, \quad \pi(m)=\frac{|\{p\in m: a(p)\in P_y\}|}{|m|}, \quad \kappa(m)=\frac{|m\cap R_y|}{|R_y|},$$

where $P_y$ is the set of class-$y$ prototypes. Purity $\pi$ penalizes masks that leak into background or into other classes. Coverage $\kappa$ penalizes masks that capture only a part of the object, which addresses the local issue. Purity cannot detect a mask that merges two adjacent objects of the same class, which is a group issue. For that case we keep SAPNet's cheap constraint and reject any mask that contains another click.

The selected mask then replaces the output of the prior-guided selector inside the released Point2RBox-v3 code, and everything else is left unchanged. As a secondary change, the rotation angle is computed from the covariance of the class-consistent pixels inside the mask rather than all mask pixels. This targets P2RBox's documented failure when masks leak into background.

We also study one refinement round, in which prototypes are re-estimated from features pooled over the selected masks. This round risks confirmation bias, so we report it as a separate condition, never as the default. Total added cost is one forward pass of a frozen encoder per training image, plus clustering.

Baselines

The naive reference point is to take SAM's highest self-predicted-IoU mask for each click and convert it to a rotated box with the eigenvector rule. This shows how much any selection rule adds at all.

The strongest existing method is Point2RBox-v3, the most direct competitor because it uses the same SAM prompts and differs only in how a mask is chosen. It reports 66.09% mAP on DOTA-v1.0 in the two-stage setting and 59.61% end-to-end. Its released code logs 61.38 mAP end-to-end and 67.24 mAP when its pseudo-labels train a rotated FCOS detector VisionXLab/Point2RBox-v3. The same paper raises the Bridge class from 8.0% to 41.6%, which shows where heuristics already help and where room remains. We run the released code and swap only the selector, keeping the same SAM checkpoint, schedule, switch epoch, detector and seeds. We reproduce its numbers first and compare against our own reproduction. Beating it means a gain over the reproduced 67.24 two-stage figure that exceeds seed variance across three seeds, together with clear per-class gains on bridge, harbor, ground track field and roundabout. A higher mean driven only by ships and vehicles would not count.

P2RBox, at 62.26% mAP with Oriented R-CNN, and Point2RBox-v2, at 62.61% two-stage, serve as reference points for the MIL-based and layout-only alternatives.

The ablations isolate the key ingredient. A single centroid per class tests the value of multiple prototypes. Removing background prototypes, and using purity alone or coverage alone, test each half of the score. Replacing DINOv3 features with SAM encoder features tests whether a separate semantic encoder matters. The linear-probe and kNN substitutions are discussed in the next section.

Why not just…?

Why not just train a linear classifier on the frozen click features and use its pixel predictions instead of prototypes? This is the most serious threat, and we do not claim prototypes must win. The file's arguments for prototypes in few-shot regimes, from Meta-Baseline and FSP-DETR, concern a handful of examples. DOTA provides thousands of clicks per common class, which is exactly where a linear probe is competitive. We therefore run the identical pipeline with a linear probe trained on click features against randomly sampled background features. Prototypes should have an edge in two places: classes with very few clicks, and rejecting pixels far from every prototype as unknown, where a linear probe forces a choice. If the linear probe matches us everywhere, the contribution narrows to "semantic per-pixel class evidence from the clicks beats geometric heuristics." We would still report that result, with prototypes as a convenience rather than the cause.

Why not just use kNN over all stored click features? kNN is the limit of many prototypes, so we sweep $M$ up to full kNN. If accuracy keeps rising with $M$, the clustering step is unnecessary and we will say so.

Why not just let the detector learn semantics during training? Point2RBox-v3's Progressive Label Assignment already switches to the network's own predictions after the switch epoch. However, early pseudo-labels set what the network learns, and errors compound through self-training, which is the confirmation bias noted in Direction 3. Measuring pseudo-label accuracy directly, as described below, separates these effects.

Why not just use the training-free ZODS-RS pipeline? It reports only 13.06 horizontal-box mAP on FAIR1M and 16.69 on xView, and it produces no rotated boxes. It is a strong matcher but a weak detector. We borrow its prototype estimation and retain a trained detector.

Proposed Experiments

The first check is a go/no-go test that requires no detector training. We hold out 20% of DOTA-v1.0 training clicks and measure nearest-prototype classification accuracy of their frozen features. If aerial classes are not separable in DINOv3 space, purity scores are noise and the project stops or switches encoders.

The decisive check uses DOTA-v1.0's full rotated boxes, which let us score pseudo-labels directly. We report the mean rotated IoU between each selected pseudo-box and its ground-truth box, per class. We also report how often each selector picks a part-level mask (local issue) or a merged mask (group issue). The prototype selector should beat Point2RBox-v3's selector most on irregular and isolated classes. If it does not raise pseudo-box IoU there, downstream mAP gains are unlikely to be real.

Downstream, we report DOTA-v1.0 mAP in both end-to-end and two-stage settings against the reproduced baseline. We add DIOR-R, which all four point-supervised papers evaluate but for which no reported number was found on Lacuna. We also add STAR, where Point2RBox-v3 logs 16.20 mAP end-to-end and the hard scenes leave room to improve.

A final stress test uses the sparse-partial regime of SPWOOD. There, many objects are unclicked, so background prototypes are contaminated; the test checks whether the method degrades gracefully. SPWOOD reports 48.5% mAP with rotated-box labels and 45.5% with horizontal boxes in its 10-10 setting, but no reported result was found on Lacuna for its point-label setting. We would therefore report our own point-label baseline there rather than quote one.