Co-occurrence Prototypes for Few-Shot Segmentation of Shapeless Tissue

Few-shot segmentation models learn to outline a new kind of object from only a handful of labeled examples. Most of them work the same way. They average the features of the labeled pixels into a single "prototype" vector, then label each pixel of a new image by how close it sits to that average. An average keeps no record of where the features were. A region that alternates two feature types pixel by pixel, and a region split into two solid halves of those same features, produce the same prototype. In natural images this loss is tolerable, because objects are defined mostly by shape. In histology and oncology imaging it is not. Malignant glands, invasive tumor, and fat pockets around implants have irregular shapes, and what distinguishes them is how local patterns are arranged.

We propose replacing the averaged prototype with a co-occurrence prototype. This is a differentiable, rotation-averaged record of which learned feature types appear next to which others, at several pixel distances, inside the labeled region. Query images are then matched to this record. The key ingredient is the displacement: statistics indexed by how far apart, and in what direction, two features sit. The proposal is built around testing whether that ingredient helps when labels are scarce. It is just as important to find where it does not help, since there is a published argument that texture is mostly noise in few-shot segmentation.

Core Claim

When the target class is defined by spatial micro-arrangement rather than outline, a prototype that encodes the joint statistics of feature pairs at fixed displacements will beat three alternatives under $K \le 5$ labeled images. The alternatives are a mean prototype, a set of local prototypes, and a displacement-free second-order prototype that encodes feature co-statistics at the same pixel. All comparisons hold dimensionality and training budget fixed. The gain should concentrate where shape cues fail: the malignant GlaS Test Part B images, and adipose clusters in SFBHI. It should be small or negative on shape-defined natural-image classes.

That predicted crossover is the point of the study. On the Texture Bias for Few-Shot CNN Segmentation suppresses high frequencies with a Difference-of-Gaussians pyramid and raises 1-shot mIoU on FSS-1000 from 75.2% to 80.8%. That work treats texture as something to remove. We expect it to be right for natural images and wrong for shapeless tissue. A design whose displacement set can be ablated lets us measure which regime a dataset is in, rather than assume it.

Relation to Prior Work

The motivating evidence comes from prototype methods.

PANet builds foreground and background prototypes by masked average pooling and labels query pixels by cosine distance. Its results are 48.1% and 55.7% mIoU at 1 and 5 shots on PASCAL-5i.

The location-sensitive local prototype network shows that on abdominal CT a single global prototype collapses to 30.3% Dice. Grid-anchored local prototypes recover 66.7%, or 70.3% with auxiliary organs.

That second result is evidence that averaging discards needed information. It is also a constraint on our design. Its fix works because healthy organs sit at stable coordinates, and glands scattered across a slide do not. Our prototype is therefore deliberately free of absolute coordinates and averaged over orientation.

DenseMP takes a different route. It lifts PA-Net on Abd-CT from 31.86% to 72.84% Dice through unsupervised dense pre-training. That changes the features, not the prototype, so it is complementary to our approach. We treat it as an orthogonal axis.

The method-supplying work comes from differentiable texture statistics.

JOSHUA inserts soft-binned spatial histogram layers into UNet skip connections and beats UNet and Attention UNet on SFBHI and GlaS. Its histograms are first-order, with no displacements, and it is trained with ordinary supervision rather than episodes.

The entropy-equipped diffusion model uses kernel density estimation to make local intensity entropy differentiable.

We reuse the soft-binning idea from both papers but apply it to pairs of positions. This yields a soft version of the gray-level co-occurrence matrix (GLCM). The GLCM is the classical table counting how often intensity $i$ appears at distance $d$ and angle $\theta$ from intensity $j$, as formalized in Chaddad et al. and contrasted with first-order statistics in Dieckmeyer et al.. Our version operates over learned feature codewords rather than raw gray levels.

SSTKD also learns structural and statistical texture modules. It does so for distillation on full Cityscapes, where a ResNet-18 student reaches 76.65% mIoU, and it never tests small samples.

The renal cancer fusion study concatenates frozen GLCM, Gabor, and other descriptors with AlexNet features for an SVM, reaching 98.54% accuracy. That is image-level classification with a two-stage, non-differentiable pipeline. We instead make co-occurrence the matching rule itself, trained end to end through episodes.

Method

A shared encoder maps an image to features $f(p)$ at each position $p$. A small learned dictionary of $M$ codewords soft-assigns each feature, giving $q(p) \in \Delta^{M}$ by a temperature softmax over cosine similarities. Here $\Delta^{M}$ denotes the set of $M$-dimensional probability vectors. For a mask $m$ and a displacement $d$ drawn from a set $D$ of distances at four orientations, the co-occurrence prototype is

$$ C_d(a,b) = \frac{\sum_p m(p)\, m(p+d)\, q_a(p)\, q_b(p+d)}{\sum_p m(p)\, m(p+d)}. $$

$C_d(a,b)$ is the probability that codeword $a$ at a masked pixel is followed by codeword $b$ at a masked pixel offset by $d$. We symmetrize it and average it over the four orientations at each distance, which gives a coordinate-free descriptor that does not depend on rotation.

At query time we compute the same quantity in a sliding window $W(x)$ around each query pixel $x$, using box filters over the $M^2$ product maps. This is affordable at $M=16$ with three distances. The query pixel's score is

$$ s(x) = \alpha \cos\big(f(x), \mu\big) - (1-\alpha) \sum_{d \in D} \mathrm{H}\big(C_d^{W(x)}, C_d^{S}\big), $$

where $\mu$ is the usual mean prototype, $C_d^{S}$ is the support prototype, $\mathrm{H}$ is the Hellinger distance between distributions, and $\alpha$ is learned.

The background gets its own co-occurrence prototype, and pixels are assigned to the closer class. Training follows PANet's episodic scheme, including its support–query alignment term.

The design deliberately contains its own controls. Setting $D=\{0\}$ reduces $C$ to a codeword co-statistic at a single position, which is a displacement-free second-order prototype. Setting $\alpha=1$ recovers PANet.

Research Program

The first stage builds the layer and verifies a basic property on synthetic textures: the soft co-occurrence prototype separates checkerboard-versus-halves pairs that mean and displacement-free prototypes cannot. The second stage defines a histology few-shot protocol, which the file shows does not yet exist in episodic form.

The main protocol meta-trains on SFBHI tissue classes plus non-gland GlaS background, then tests on GlaS glands with $K \in \{1, 3, 5\}$ support images drawn from the 85-image training split. Test Parts A and B are reported separately. The reverse direction meta-trains on GlaS and tests on SFBHI adipose.

Abdominal CT, following DenseMP's Abd-CT protocol, and FSS-1000 serve as shape-defined controls. On these we predict at most parity.

Every method is run on two backbones. One is an ImageNet ResNet-50. The other is a frozen pathology foundation encoder of the kind MPAMatch uses (UNI). The second backbone tests whether the question survives strong pretrained features.

Stain and scanner sensitivity is a known weakness of texture statistics. We probe it by training with and without stain normalization and by testing under held-out color perturbations.

Baselines

The naive reference is a UNet trained directly on the same $K$ labeled images with AugMix, elastic warping, and affine augmentation. It gets the same epochs and the same hyperparameter search budget, about 20 configurations per method chosen on a held-out validation split of SFBHI or GlaS training images, never on the test sets. AugMix is included because the synthetic-data limitations study found it matched or beat diffusion and GAN augmentation in data-scarce domains. It is therefore the honest "just augment" bar.

The direct competitor is PANet with masked average pooling on the identical backbone and episodes. It differs from our method only in the prototype, which makes it the cleanest comparison.

The spatial-prior competitor is the location-sensitive local prototype network, run with overlapping 1/8 grids as in its paper. It reports 70.3% Dice on VISCERAL. We expect it to stay strong on Abd-CT and weaken on GlaS, where gland positions are arbitrary.

On Abd-CT, the strongest reported few-shot number in the file is DenseMP plus PA-Net at 72.84% mean Dice. We will plug our prototype into DenseMP-pretrained features to test additivity.

On GlaS, the strongest reported number is MPAMatch at 92.44% mDice, against UniMatch at 91.22%. Both are semi-supervised and use the full training split plus foundation models. Neither is a $K$-shot result, so we report them as a data-rich ceiling. On FSS-1000, DoG-LSTM's reported 80.8% 1-shot mIoU is the competitor. We expect to be at or below it there.

The ablation that isolates the key ingredient is our own model with $D=\{0\}$, at matched codeword count and parameter count. A second ablation matches the dimensionality of $M^2|D|$ with $|D|$ mean prototypes obtained by clustering the support features.

Beating the field means three things. On GlaS Test Part B, at 3 and 5 shots, we must exceed PANet, local prototypes, and the $D=\{0\}$ ablation by a margin outside the across-episode 95% confidence interval, on both Dice and the challenge's object-level F1. The gain on Part B must be larger than on Part A. The model must not lose more than a point on Abd-CT relative to the same backbone's PANet.

Why not just…?

Why not just augment harder? Augmentation enlarges the support set but leaves the matching rule unchanged. The averaged prototype still cannot represent arrangement. The COVID chest X-ray study shows that standard transfer and preprocessing on small cohorts still learned shortcuts, with sensitivity of 0.54 on CORDA and a collapse across datasets. This remains a real threat in segmentation, though, because heavy elastic and color augmentation might regularize a UNet enough. The naive baseline is tuned with the same budget as our method precisely so that it can win if it deserves to.

Why not just use a pathology foundation model with PANet? This is the strongest threat. Foundation features already encode local texture within each patch, so displacement statistics might add nothing. The file gives partial evidence that they leave room: MPAMatch needed structured prototype alignment on top of UNI to move from 91.22% to 92.44% and to fix merged adjacent glands. But that result is not a $K$-shot test. Our two-backbone design settles the question directly. If the co-occurrence gain over PANet vanishes on the frozen foundation encoder, we will report that the ingredient only matters for weaker features.

Why not just use richer pooling without displacements, such as covariance pooling or several clustered prototypes? These capture variance within a region, which is part of what averaging loses. Whatever remains to be gained must therefore come from spatial arrangement. CNN receptive fields already encode some arrangement, so this alternative could plausibly match us. That is why the $D=\{0\}$ and clustered-prototype ablations are the central experiments rather than side checks. If they tie the full model, the claim about displacements is false and the contribution reduces to second-order pooling.

Why not just use handcrafted GLCM features with an SVM or random forest? Fixed descriptors avoid overfitting, and the renal study shows they add value when fused with deep features. That study classifies whole images, however. The GlaS challenge reports that its top seven methods were CNNs, and the gap was largest on malignant tissue. We still include a per-pixel GLCM random forest, computed on windowed raw intensities, as a reference at every $K$, because it may be competitive at $K=1$.

Decisive Checks

The study succeeds or fails on four outcomes. First, on GlaS Part B, the full co-occurrence prototype must outperform its own displacement-free ablation. Second, that margin must be larger on Part B than on Part A. Third, the margin must persist on the frozen foundation backbone. Fourth, on FSS-1000 the method must show no gain, or a loss, relative to DoG-LSTM's 80.8%.

If all four hold, the "texture as noise" and "texture as signal" positions are reconciled empirically: displacement statistics help when classes lack stable shape and hurt otherwise. If the displacement-free ablation ties the full model, the result is still useful. It would show that what few-shot prototypes lack is second-order statistics, not spatial arrangement, and that would redirect work toward cheaper pooling.