Vision foundation models trained on natural photographs transfer well to medical scans and satellite imagery. They carry a built-in resolution loss, though. They cut each image into fixed 14×14-pixel tiles ("patch tokens") before any reasoning happens, so every downstream decoder has to reconstruct fine boundaries from a coarse grid. Irregular, indented targets suffer most from this: organs wrapped around other organs, winding hedgerows seen by a 10 m satellite, curved text. The usual response is to spend more compute, either by feeding a larger image, adding a high-resolution branch, or attaching a heavier upsampling decoder.
We propose a cheaper route. Each coarse token does not predict which of its pixels belong to the object. Instead, it predicts a few boundary points in continuous image coordinates, as offsets from its own center. The object outline is then the concave hull (a tight, indented envelope, as opposed to a rubber-band convex hull) of all the predicted points. How finely the boundary is placed then depends on regression accuracy, not on the token grid. The central claim is that with a frozen or lightly adapted natural-image backbone, this point-set output matches the boundary accuracy of a mask decoder running at twice the input resolution, at roughly a quarter of the backbone compute. It also avoids the fragmentation and merging failures that make concave hulls brittle.
What exists and what is missing
Three strands supply the ingredients.
The first is dense boundary-point regression wrapped by an alpha shape. An alpha shape is the concave hull obtained by deleting Delaunay triangles whose circumradius exceeds a radius α, as described in Impacts of Catchments Derived from Fine-Grained Mobility Data on Spatial Accessibility. This combination is MSR: Multi-Scale Shape Regression for Scene Text Detection: pixels inside a text core regress offsets to the nearest boundary point, and an alpha shape closes the polygon. MSR reports an F-score of 81.5% on CTW1500, compared with 60.4% for a quadrilateral-box baseline. MSR supplies our output representation. It differs from our setting in three ways: it is trained from scratch, it fights resolution loss by running several full-resolution image scales through the network, and it reports that a single global α merges neighboring instances and fractures sparse ones.
The second strand is hull-shaped detection in crowded scenes. Beyond Bounding-Box: Convex-Hull Feature Adaptation wraps nine predicted points per object in a convex hull and penalizes points shared between instances. It reaches 76.67% mAP on DOTA. This work motivates our instance-separation mechanism and constrains our design: its authors note that convex envelopes bridge over concavities, which is exactly the failure we target.
The third strand is transfer of natural-image features into specialized domains. DINOv2 based Self Supervised Learning For Few Shot Medical Image Segmentation adapts DINOv2-large with low-rank adapters (LoRA) and a slice adapter for abdominal CT/MRI. It reaches 73.72 and 78.43 mean Dice, but it decodes masks from the 14-pixel token grid and cleans them with connected-component filtering. Hedgementation shows that frozen foundation embeddings with simple classifiers produce fragmented hedgerow masks, with IoU around 11–23%. A temporal U-Net reaches just over 40%. FM-OSD and Natural Image Bases to Represent Neuroimaging Data motivate keeping a natural-image backbone rather than training in-domain.
None of these works regresses sub-patch boundary geometry from foundation tokens. None of them uses a concave hull to turn that geometry into a polygon.
Core Claim
Coarse transformer tokens carry enough local information to locate a boundary within their own tile. Mask decoders throw this information away when they force a per-pixel decision through interpolation. If each token instead emits $K$ boundary points $p = c + \Delta$, where $c$ is the token center and $\Delta$ a predicted continuous offset, then boundary error should be set by offset-regression noise rather than by the stride.
The concave hull stage becomes reliable if it has two properties that MSR lacked. The first is a local radius: each point's α is set from the spacing of its own neighbors, so dense regions do not fragment and sparse regions still carve indentations. The second is instance or class tags inherited from the emitting token, so the hull is built per instance and never across instances. The hull stage itself stays non-differentiable. We train the point geometry so that the hull is well-posed, and we test whether that suffices rather than assuming a differentiable hull is needed.
Research Program
The backbone is DINOv2 with the LoRA and slice-adapter recipe of the few-shot CT/MRI paper. For satellite data the same recipe is applied, with a band adapter mapping Sentinel-2 channels to three inputs.
On top of the backbone sits a small head with three outputs per token. The first is a foreground or class score. The second is $K$ offsets, with $K$ between 4 and 16, which may extend up to two tiles away so that thin structures narrower than a tile can still be outlined by neighbors. The third is an instance embedding, used only where a class has multiple instances, such as text lines or hedgerow segments. In the few-shot medical setting, the head takes the support–query similarity map alongside query features and is trained class-agnostically on training organs.
Points are trained with a two-sided distance loss. The loss lets predicted points slide along the true contour, following the unsigned-distance idea in Joint Dense-Point Representation for Contour-Aware Graph Segmentation, and also forces the contour to be covered:
$$ \mathcal{L}_{pts} = \frac{1}{|P|}\sum_{p\in P} d(p,\partial G) + \frac{1}{|B|}\sum_{b\in B}\min_{p\in P}\|p-b\|, $$
Here $P$ is the set of predicted points from foreground tokens, $\partial G$ is the ground-truth boundary, $B$ is a dense sample of points on it, and $d$ is the distance to the boundary. The first term keeps points on the contour and the second prevents gaps, which are the root of hull fracturing.
At inference, the local radius for each point is $\alpha_p = \lambda \cdot r_k(p)$, where $r_k(p)$ is the distance to the $k$-th nearest predicted point with the same tag, and $\lambda$ is a single constant tuned on validation data. A variant with a learned radius is an ablation.
Baselines
The naive reference point is the published DINOv2 few-shot pipeline run unchanged on Abd-CT and Abd-MRI: a mask from the token grid, bilinear upsampling, and connected-component cleanup. The paper reports 73.72 and 78.43 mean Dice, respectively. No reported boundary-distance result for it was found on Lacuna, so we will rerun it and measure 95th-percentile Hausdorff distance and average surface distance ourselves.
The strongest competitor is the same backbone with a mask decoder fed an upscaled input at 1.5× and 2×. This shrinks the effective patch to about 9 and 7 pixels, and it is the obvious way to buy resolution. A second strong competitor adds the distance-map boundary loss of Boundary loss for highly unbalanced segmentation to that decoder. That loss raised ISLES Dice from 0.575 to 0.656 and cut Hausdorff distance by 10% on WMH, so it is the best existing way to sharpen boundaries without changing the output form.
All models share the same backbone, the same LoRA rank, a matched head parameter count, identical training steps, and the same hyperparameter search budget. We report backbone FLOPs alongside accuracy. Beating the upscaling baseline means matching or exceeding its Dice and Hausdorff distance at 2× while running our head at 1× input, or clearly winning at equal FLOPs.
On Hedgementation, the reference is the reported U-TAE at just over 40% IoU (45.9% on agricultural pixels). The foundation-embedding reference is AEF with a Random Forest, at about 22–23%. Our comparison is against a DINOv2 mask decoder under the same adapter. On CTW1500, MSR's 81.5% F is the reference for whether a foundation backbone with a point head can replace multi-scale from-scratch training.
Four ablations isolate the key ingredients. The first swaps our head for a mask head at equal capacity. The second uses a convex hull per instance in the style of CFA. The third uses a global α. The fourth drops the instance tags.
Why not just…?
Why not just feed the backbone a larger image? This is the most serious threat. It is simple, and it reduces the effective patch size directly. Its cost is roughly quadratic in the scale factor for the attention layers. The decisive experiment is to plot boundary error against effective stride for both heads. If the mask head's error falls with stride while ours stays flat at 1×, the point representation earns its place. If upscaling at 2× beats our 1× head at equal FLOPs, the claim fails and we will report that.
Why not just add a boundary loss to a mask decoder? A boundary loss sharpens where the mask edge falls, but the mask is still decided on an interpolated coarse grid. The mask-plus-boundary-loss baseline at matched compute tests this directly. A small gap would mean the gain comes from supervision rather than representation, and we would say so.
Why not just clean masks with connected components or morphology? As the DINOv2 few-shot work itself uses it, connected-component filtering removes islands but cannot move a leaking edge or recover a thin structure that was never activated. It is already part of the naive baseline, so any gain we show is on top of it.
Why not put learnable pooling into the tokenizer instead? LIP improves small-object AP on COCO from 21.9 to 24.0, but it requires retraining the downsampling layers. Retraining the patch embedding of a natural-image foundation model forfeits the very pretrained features whose transfer the few-shot and FM-OSD results depend on. We keep the tokenizer frozen by design and will not test a retrained tokenizer.
Why not use convex hulls, as CFA does? Convex hulls avoid fragmentation but fill concavities. Organs such as the liver around the gallbladder, curved text, and branching hedgerows are exactly the cases where they fail. The convex-hull ablation measures how much of the boundary error concavity accounts for.
Proposed Experiments
The primary test is few-shot organ segmentation on Abd-CT and Abd-MRI under the published protocol. We report Dice, 95th-percentile Hausdorff distance, and average surface distance, broken down by organ concavity (the ratio of convex-hull area to mask area). The prediction is that gains grow with concavity.
The second test is Hedgementation. There we evaluate against the original BD Haie vector lines, not only the rasterized 10 m labels, because rasterized labels cannot reward sub-pixel boundary placement. We report both IoU and line-distance error, and check whether fragmentation, measured as the number of components per true hedge, drops relative to the mask decoder and the AEF baselines.
The third test is CTW1500 against MSR. It probes the instance-merging fix, and we report merge and split counts.
The single decisive check is the error-versus-stride curve described above. If boundary error for the point head is not flatter in stride than for the mask head, the core claim is wrong regardless of Dice.
References
- Impacts of Catchments Derived from Fine-Grained Mobility Data on Spatial Accessibility (Short Paper)
- MSR: Multi-Scale Shape Regression for Scene Text Detection
- Beyond Bounding-Box: Convex-Hull Feature Adaptation for Oriented and Densely Packed Object Detection
- DINOv2 Based Self Supervised Learning for Few Shot Medical Image Segmentation
- Hedgementation = Hedgerow Segmentation: A Remote Sensing Benchmark
- FM-OSD: Foundation Model-Enabled One-Shot Detection of Anatomical Landmarks
- Natural Image Bases to Represent Neuroimaging Data
- Joint Dense-Point Representation for Contour-Aware Graph Segmentation
- Boundary loss for highly unbalanced segmentation
- LIP: Local Importance-Based Pooling