Keeping Clusters Honest About Context: Why Joint Clustering Breaks Once the Receptive Field Goes Global

Unsupervised semantic segmentation means labeling every pixel of an image with a category, such as road, sky, or person, without ever seeing a labeled example. The standard recipe groups per-pixel features into clusters. The best current systems take a large pretrained vision transformer, a network whose every output can look at the whole image, freeze it, and train only a small head on top. That choice contradicts a decade of deep-clustering work arguing that the feature extractor should be trained jointly with the clustering objective. The segmentation authors report that joint training produced blurry maps and missed objects, so they gave it up. This note proposes an explanation for that failure and a targeted fix.

The explanation is that a network with a global receptive field has a shortcut available when it is trained to make pixel clusters confident and consistent. It can let every pixel's cluster be decided by image-level context: "this is a beach scene" rather than "this pixel is sand." Clusters then encode scenes instead of regions, and boundaries smear. Older jointly trained segmenters built on ordinary convolutional networks escaped this shortcut by accident, because such networks are nearsighted. The fix we propose follows from the diagnosis. During joint training, we penalize how much a pixel's cluster assignment depends on pixels the model itself currently places in other clusters. We allow a small, bounded amount of cross-region context. The backbone stays free to reshape under clustering pressure, but it cannot route that pressure through the context shortcut.

Core Claim

The claim has three linked parts, each testable.

First, the locality of convolutional networks acted as an implicit regularizer for joint pixel clustering. Understanding the Effective Receptive Field showed that the region of input that actually influences a convolutional output decays like a Gaussian and grows only as the square root of depth. It also showed that residual connections shrink that region further. PiCIE trained a ResNet jointly with a pixel-clustering loss, combining invariance to color jitter with equivariance to crops and flips. Under our hypothesis, it could do so stably precisely because each pixel could barely see beyond its neighborhood.

Second, giving that same joint pipeline a global receptive field will reproduce the collapse that STEGO reported. This holds whether the global view comes from an efficient attention block like the row-and-column Criss-Cross Attention or from an unfrozen transformer. The collapse should show up as a measurable rise in how much each pixel's assignment depends on pixels outside its own segment.

Third, capping that out-of-segment dependence, rather than capping how far the weights move, makes joint fine-tuning of a global backbone stable. It then beats the frozen-backbone systems.

To make "dependence on other segments" precise, let $a_i(x) \in \Delta^K$ be the soft assignment of pixel $i$ over $K$ clusters for image $x$. Let $\hat{c}(j)$ be the current hard cluster of pixel $j$. The diagnostic, measured with the gradient method of the receptive-field paper, is the leakage

$$\Lambda_i = \frac{\sum_{j:\hat{c}(j)\neq \hat{c}(i)} \|\partial a_i / \partial x_j\|}{\sum_{j} \|\partial a_i / \partial x_j\|},$$

the fraction of pixel $i$'s input sensitivity that falls on pixels assigned to other clusters. Computing this during training is too expensive, so the training penalty uses a cheaper proxy.

In a transformer, one extra forward pass can use a block-diagonal attention mask, so that each image patch attends only to patches currently in its own cluster. This gives a segment-restricted assignment $\tilde{a}_i(x)$ with stop-gradient on the mask. The regularizer is

$$\mathcal{R} = \frac{1}{N}\sum_{i=1}^{N} \max\big(0,\ \mathrm{KL}(a_i(x)\,\|\,\tilde{a}_i(x)) - \tau\big),$$

where $N$ is the number of patches and $\tau$ is a budget on how much cross-region context may change a pixel's cluster. The hinge matters. Context is genuinely needed, for instance to tell a boat from the water around it, which is PiCIE's documented failure. So we allow context up to $\tau$ rather than banning it. For convolutional backbones fitted with criss-cross attention, the same mask is applied inside the attention step.

Research Program

The work has two stages, and the first can falsify the premise before any method is built.

The first stage is a diagnostic study. We reproduce PiCIE on its ResNet backbone and record leakage $\Lambda$ on held-out images throughout training. We then insert two recurrent criss-cross attention blocks after the last stage and retrain with the identical loss. As a third arm, we unfreeze the self-supervised DINO ViT-S/8 transformer that STEGO uses and train a PiCIE-style clustering loss through it.

Our hypothesis predicts three things. Leakage should stay low for the plain ResNet. It should rise in both global arms. The rise should precede or coincide with the drop in segmentation quality and with a measurable change in what the clusters encode. We quantify that last point by predicting each image's scene label from its cluster histogram: if the clusters are turning into scene detectors, that prediction should get easier.

If leakage does not rise, or rises without any loss in quality, the premise is wrong. In that case we would report the negative result rather than proceed.

The second stage is the method. We fine-tune the DINO transformer jointly with a clustering objective and add $\mathcal{R}$. For the clustering objective we use STEGO's correspondence-distillation loss, so that the only change from STEGO is unfreezing plus the regularizer. Balanced assignments come from the entropic optimal-transport step of Differentiable Deep Clustering with Cluster Size Constraints. That paper uses marginal constraints to rule out the all-in-one-cluster collapse, and we reuse its method for exactly that purpose. We relax the uniform marginal toward an estimated class prior, because COCO-Stuff is dominated by background "stuff" pixels. The same regularizer is also applied to the PiCIE-plus-criss-cross-attention model. This tests whether the fix is specific to transformers or general to any global-context backbone.

Prior work plays three different roles here. The receptive-field analysis and PiCIE motivate the hypothesis. STEGO's report of joint-training failure constrains the design, since any proposal has to survive the regime where it failed. The optimal-transport assignment supplies part of the method. PriMaPs proposes masks from principal components of frozen features and fits prototypes with expectation-maximization. It is complementary rather than competing, because it never updates the backbone, and it can be layered on our output exactly as its authors layer it on STEGO.

Baselines

The naive reference point is k-means run directly on frozen DINO ViT-S/8 patch features. PriMaPs reports this family's DINO ViT-S/8 baseline at 15.8% mIoU on Cityscapes. It shows how much any trained head or fine-tuning adds at all.

The strongest existing method is STEGO on frozen DINO. It reports 28.2 mIoU on 27-class COCO-Stuff and 21.0 mIoU on Cityscapes. We also report PriMaPs-EM layered on STEGO, at 29.7% mIoU on COCO-Stuff, since we can apply the same layering to our model. These are the strongest competitors because they share our backbone and our target benchmarks. They also embody the opposing position that freezing is necessary.

For fairness we rerun STEGO from the released configuration, on the same ViT-S/8, the same training images, and the same CRF post-processing. We give it the same number of hyperparameter trials we spend on our own method. Its GPU-hours are matched by extending its head training to our fine-tuning budget. Beating it means exceeding our own rerun, not only the published number, by more than the seed-to-seed spread across three seeds on both COCO-Stuff and Cityscapes. We also report Potsdam-3, the aerial-imagery benchmark STEGO and PriMaPs evaluate on, where no reported result was found on Lacuna for either system. For that dataset we rely only on our reruns.

The convolutional baselines are PiCIE, which reports 65.5% pixel accuracy on Cityscapes with no mIoU found on Lacuna, and PiCIE with criss-cross attention inserted. The latter is the "obvious" combination at this intersection and is tested with and without $\mathcal{R}$.

Two ablations isolate the key ingredient, which is that the constraint is conditioned on clusters. The first replaces $\mathcal{R}$ with an L2 penalty pulling weights toward their pretrained values, tuned to the same distance from initialization. This tests whether we are just fine-tuning gently. The second keeps the masked-attention mechanism but builds the mask from a random partition of patches, and separately from fixed spatial windows. This tests whether any attention restriction works, or whether restriction to the model's own clusters is what matters. We also sweep $\tau$ from zero, meaning no cross-region context allowed, to effectively unbounded, meaning plain joint training.

Why not just…?

Why not just keep the backbone frozen, as STEGO and PriMaPs do? This is the default, and it is strong. The case against it is that its known errors are exactly the ones a fixed feature space cannot repair. PriMaPs confuses trucks with buses, and STEGO merges food objects with food surfaces and ceilings with walls. A head trained on frozen features can only redraw boundaries within the geometry it inherits. The head-to-head comparison above, under matched compute, settles it. If regularized joint training does not beat our STEGO rerun, freezing wins and we say so.

Why not just fine-tune gently, with a small learning rate, a few unfrozen top blocks, or a moving-average teacher? This is the real threat. If joint training fails simply because the weights drift too far, any proximity constraint would do. The L2-to-initialization ablation, matched in how far the weights move, is designed to answer this. So is a variant that unfreezes only the last two blocks. Our hypothesis predicts that gentle fine-tuning either stays near STEGO's performance or still shows rising leakage and scene-like clusters. If it matches our method, the context-shortcut story is unnecessary, and the contribution reduces to showing that careful fine-tuning suffices.

Why not just add criss-cross attention or large kernels to PiCIE to fix its nearsightedness? Two reasons argue against this. First, the global-receptive-field transformer used by STEGO already exceeds PiCIE by the 14 mIoU on COCO-Stuff that STEGO reports. Widening a ResNet's view therefore competes with a stronger, readily available alternative. Second, our hypothesis predicts that this combination will actually destabilize PiCIE's joint training. The diagnostic stage tests that prediction directly, and it is informative either way.

Why not just clean up the maps with a conditional random field? A random field only snaps labels to local color edges. As the PiCIE and STEGO failure cases show, it snaps a wrong "water" label neatly onto a boat's outline. All systems in our comparison receive the same random-field step, so it cannot account for any difference we report.

Proposed Experiments and Decisive Checks

We evaluate on 27-class COCO-Stuff, Cityscapes, and Potsdam-3 using the standard protocol of Hungarian matching between clusters and classes. We report mIoU and pixel accuracy over three seeds, plus mIoU restricted to "thing" classes, where frozen features fail most.

The first decisive check is whether leakage $\Lambda$ separates the stable local model from the unstable global ones, and whether it predicts the quality drop across runs. We also check whether the scene-label probe on cluster histograms becomes more accurate as joint training degrades.

The second is whether regularized joint fine-tuning beats our STEGO rerun on COCO-Stuff and Cityscapes beyond seed variance. It should also beat the L2-to-initialization and random-mask ablations. Losing to either ablation would reject cluster conditioning as the active ingredient.

The third is whether the gains concentrate on the documented confusions. We check the truck–bus pair on Cityscapes and the food and ceiling–wall pairs on COCO-Stuff using per-class-pair confusion rates. This is where reshaping the backbone should matter and where a better head alone should not.

Finally, the $\tau$ sweep should trace an inverted U. Performance should be low with no cross-region context, because boats and water become inseparable, and low with unlimited context, because clusters become scene detectors. It should peak in between. A monotone curve in either direction would indicate that the context budget is not the mechanism we think it is.