Uncertainty-Consistent Set Matching for Touching, Low-Contrast Instances

Many medical images contain several objects that touch across faint edges. Cell nuclei crowd each other in a pathology slide, and a lesion blends into the tissue around it. Models that predict objects as an unordered set handle this in two steps. First they assign each prediction slot to one ground-truth object by solving a matching problem (the Hungarian algorithm finds the cheapest one-to-one pairing of predictions to targets). Then they train each slot against the object it was assigned. The matching cost treats every pixel of an object as equally trustworthy, including pixels on a boundary the image barely supports. When two nuclei meet along a blurry interface, small changes in those unreliable pixels can flip which slot is assigned to which nucleus from one training step to the next. Each slot is then trained toward a different object depending on the step.

We propose to make the matching cost and the training loss the same probabilistic quantity. Each slot predicts two things: a signed-distance map describing its object's shape (negative inside, positive outside), and a per-pixel uncertainty describing how sure it is of that distance. Pairs are matched by the likelihood of the target shape under the slot's prediction. Pixels on faint boundaries then count less in deciding which object a slot owns, while confident interiors dominate. The same likelihood trains the shape, so matching and learning never disagree. The signed-distance representation has no star-shape restriction, so concave and elongated objects are handled too. The core bet is that, under low contrast, the main failure of set prediction is unstable identity assignment rather than poor boundary pixels, and that fixing the cost fixes both.

Core Claim

The only existing system that joins bipartite matching with tolerant boundary supervision is LSP-DETR. It segments whole-slide nuclei by having each query predict a center and 64 radial distances, and it trains those distances with a "flexible" interval loss. Its authors state that nuclear borders remain hard in poorly focused or very low-contrast regions. Their tolerance is a fixed geometric interval that does not read the image, and it enters the loss rather than the matching cost. It is also bound to star-convex shapes.

The boundary-loss literature, meanwhile, weights edge pixels with fixed rules. Distance Map Loss multiplies cross-entropy by an inverted distance transform. Contour Loss puts focal weights on perimeter pixels. Polyper splits a coarse mask into interior and boundary bands by fixed erosion and dilation. All of these operate on a single fused foreground map with no instance identity. Polyper's authors themselves list the fixed band width as a limitation.

Our claim differs from both lines in one specific way. The uncertainty is learned per slot and per pixel, and it sits inside the Hungarian cost. The hypothesis is that this lowers the rate at which assignments flip during training on touching, low-contrast instances, and that this lower flip rate is what drives the gains in instance quality. We will measure that link directly rather than infer it from final scores.

Method

Each of the $N$ query slots outputs a class distribution $p_i$, a truncated signed-distance map $s_i(x)$, and a positive scale map $b_i(x)$ over pixels $x$ in a window around the slot. Each target $j$ has a truncated signed distance $d_j(x)$, clipped at radius $R$ so that far-away pixels do not dominate. The pairwise term is a Laplace negative log-likelihood of the target shape, plus a class term:

$$L_{ij} = -\lambda \log p_i(c_j) + \frac{1}{|W|}\sum_{x \in W} \left[ \frac{|s_i(x) - d_j(x)|}{b_i(x)} + \log b_i(x) \right]$$

Here $c_j$ is the target class, $W$ is the union of the two objects' windows, and $\lambda$ balances class against shape. The Hungarian cost is $C_{ij} = L_{ij}$ with gradients stopped. The training loss is $L_{i\hat\sigma(i)}$ summed over the matched pairs $\hat\sigma$, with unmatched slots trained toward the empty class.

The $\log b$ term stops a slot from claiming high uncertainty everywhere to win matches cheaply. Inflating $b$ lowers the first term but raises the second, so at the optimum $b$ tracks the slot's actual error. That error should be largest exactly where the image gives weak evidence. The predicted mask is $\{x : s_i(x) < 0\}$, which places no star-convexity constraint on the shape.

The scale head reads decoder features, so it can respond to local contrast. This is the "adaptive tolerance width" that the fixed-kernel methods lack.

Research Program

We will build this as a replacement matching head and loss on the LSP-DETR design: the same lightweight linear-attention transformer and the same queries, with only the output representation and the cost changed. Any difference can then be attributed to matching and representation rather than to the backbone.

The analysis has three parts, run in order. First, we log the assignment $\hat\sigma$ for a fixed probe set of training images at every epoch. We compute the flip rate, meaning the fraction of targets whose assigned slot changes between checkpoints, stratified by boundary contrast. Contrast is measured as the mean ground-truth-normal intensity gradient along each instance's boundary. Second, we check whether $b$ is calibrated, testing whether predicted scale correlates with realized boundary error on held-out data. Third, we separate the two effects of the design, uncertainty in the cost versus uncertainty in the loss, using the ablations described below.

Baselines

The naive reference point is the same architecture and signed-distance output trained with a conventional DETR-style cost and loss. That means class probability plus Dice and binary cross-entropy on the thresholded mask, with every pixel weighted equally in both the cost and the loss. This shows what set prediction with arbitrary shapes gives before any boundary handling. It is run with identical data, schedule, and query count.

The strongest existing competitor is LSP-DETR. It reports a multi-class Panoptic Quality (mPQ) of 48.2 on PanNuke, and it is the only method in this space that already combines bipartite matching with tolerant boundaries on crowded nuclei. We will retrain it on our splits with its published settings and the same tuning budget we give our method, measured in total GPU-hours and number of hyperparameter trials. The reported 48.2 serves as a sanity check on our reproduction. Beating it means a higher mPQ on PanNuke averaged over three seeds, with the gain concentrated in the low-contrast and low-solidity (concave) strata. A tie overall combined with clear wins in those strata would still support the claim. A loss in those strata would refute it.

For MoNuSeg we found no reported result on Lacuna for LSP-DETR, so we compare only against our own retrained runs.

Two ablations isolate the key ingredient. In the first, the Laplace likelihood is used only in the loss, and matching still uses the equal-weight cost. This tests whether putting uncertainty in the cost matters. In the second, the learned $b$ is replaced by a fixed boundary band, weighting pixels by an inverted distance transform in the spirit of Distance Map Loss. This tests whether learning the tolerance matters beyond a hand-set tolerance.

A third baseline replaces hard Hungarian matching with entropy-regularized soft matching (Sinkhorn-style), since that is the standard fix for assignment instability. It gets the same tuning budget.

Why not just…?

Why not just use a strong semantic segmenter with a boundary loss? For single-object images this is a real threat. On polyp benchmarks, which mostly contain one lesion per image, set matching adds nothing, and Polyper reports 86.51% mean Dice on ETIS with a fused mask. We therefore do not claim gains there. The claim is limited to crowded, touching instances. There a single fused map merges neighbors and needs watershed-style post-processing, which LSP-DETR reports slows pipelines to 2.38 s/mm² for HoVer-NeXt versus 0.46 s/mm² for set prediction on TCGA. The PanNuke comparison settles this regime.

Why not just add a boundary loss after standard matching? This is the most plausible competitor. It is exactly the loss-only ablation above. If it matches the full method in both mPQ and flip rate, then the cost does not matter and our central hypothesis is wrong. We would report that outcome.

Why not just keep LSP-DETR's star-convex rays? Rays cannot represent concavities. LSP-DETR's authors also note degradation under blur and low contrast, and its tolerance interval does not read the image. The low-solidity and low-contrast strata test both limits separately.

Why not just use soft matching to stabilize assignments? Soft matching smooths assignments uniformly. It does not know which pixels are unreliable, and it can blur identity between touching neighbors. We include it as a baseline. If it lowers the flip rate as much as our method at equal mPQ, the uncertainty-weighted cost is unnecessary.

Proposed Experiments

The primary benchmark is PanNuke (19 tissue types, crowded nuclei), evaluated by mPQ. We stratify its instances by boundary contrast quartile and by solidity, where solidity below a threshold marks non-star-convex nuclei. MoNuSeg tests cross-organ generalization.

For a controlled stress test, we apply graded Gaussian blur and contrast reduction to the PanNuke test images. We then plot mPQ and flip rate against degradation strength for every baseline. The method should degrade more slowly than both LSP-DETR and the loss-only ablation.

As a secondary check that the tolerance head helps boundaries even without instance structure, we transplant the Laplace boundary loss onto a single-slot model for ETIS and CVC-ColonDB. We compare against published fused-map results there (Polyp-PVT reports 0.808 mean Dice on ColonDB and 0.787 on ETIS; Polyper reports 86.51% on ETIS) and against a retrained version of the same backbone. We expect parity rather than a win.

The decisive check has three parts that must hold together. The full method must lower the flip rate relative to the loss-only ablation, specifically in the lowest-contrast quartile. That reduction must precede and predict the mPQ gain in that quartile. And $b$ must correlate with held-out boundary error. If the mPQ gain appears without the flip-rate reduction, the mechanism we propose is not the one at work, and the note's central claim fails.