Testing Open-Vocabulary Masks on the Image That Proposed Them

Vision-language models such as CLIP can be asked to find any named concept in an image. You score every image patch against a text prompt and keep the highest-scoring region. The trouble is that they always find something. Prompt "giraffe" on a kitchen photo and the model still returns its best-matching blob, and nothing in the output says whether that blob is evidence or noise. The usual way to tell is a statistical test, but the region was chosen to look as strong as possible, so testing it on the same pixels is "double-dipping" and badly overstates significance. The textbook fix is to split the data: choose on one part, test on the other. For a single image, that means either collecting a held-out set of concept-free reference images or throwing away half the patches. Both impose a data tax.

This note proposes a split-free test that uses the image's own remaining patches as the null reference. It makes that reuse valid by randomizing which region is selected, a technique from differential privacy, and pays a known, closed-form price for doing so. The proposal also makes a specific claim about why this is hard for CLIP. A known defect of CLIP's patch features, a large image-wide component that drowns out local detail, is exactly what makes the randomization expensive. Removing that component with existing training-free surgery should therefore buy statistical power, not just nicer-looking maps.

Core Claim

Recent analysis of frozen CLIP explains its weakness at dense prediction (labelling individual pixels or patches). ClearCLIP shows that the final residual connection carries a few high-norm channels that broadcast the same global signal to every patch. CLIP Tricks You shows that the text embedding spends about 68% of its final-token attention on the uninformative start token. As a result, background patches can score higher against a prompt than the true object. We call the mismatch between the image-level vector CLIP was trained to align and the patch-level vectors dense tasks need the global-local alignment gap.

Statistically, this gap has two consequences. The first works in our favour. The broadcast component is shared by every patch of an image, so a contrast between a selected region and the rest of the same image is largely insensitive to it. A within-image null therefore needs no separate calibration images. The second is the cost. Cosine scores are normalized, so a dominant shared component compresses how much patch scores can differ from one another.

That compression matters because of how the selection is randomized. We use the exponential mechanism: the model draws a region with probability proportional to the exponential of its contrast score scaled by a temperature $\varepsilon$, rather than taking the top-scoring one. The selection's dependence on the data is then bounded by a stability parameter. For contrast scores confined to a fixed range of width $B$, that parameter is $\eta = \varepsilon B$, and this bound holds regardless of spatial dependence. By Theorem 1.1 of Post-selection inference via algorithmic stability, a test run at level $\alpha e^{-\eta}$ on the selected region keeps error at most $\alpha$.

Because $B$ must be fixed before seeing the data, compressed contrasts force a choice between a large $\varepsilon$ and a heavy level correction, or a small $\varepsilon$ and a near-random choice of region. Our central prediction follows. Removing the shared component, by applying ClearCLIP surgery or by projecting out the global token direction, enlarges contrasts relative to $B$. That should raise power at a fixed false-alarm level, with no reference images held out and no patches discarded.

Relation to Prior Work

Three papers motivate the problem and also mark out what we will not do. Each tests whether a selected region differs from background using exact conditional selective inference: computing the null distribution of the test statistic conditional on the selection having come out as it did.

Computing Valid P-Values for Image Segmentation by Selective Inference does this for graph cuts and thresholding under Gaussian noise. Quantifying Statistical Significance of Neural Network-based Image Segmentation extends it to piecewise-linear networks by tracing a one-dimensional path through input space. Statistical Test for Attention Maps in Vision Transformers handles softmax attention with a Lipschitz-bounded grid search.

All three test raw pixel intensity in single-modality models and must trace the network's decision geometry. We instead test a semantic hypothesis, that the prompt matches this region better than the rest of the image. We never trace the network: the network is treated as a fixed feature map, and only the final region choice is randomized. The cost is a different and weaker guarantee. Our guarantee is about the patch-score field and rests on a spatial-dependence assumption, whereas theirs is exact under Gaussian pixel noise.

Inference post region selection (POSIR) gives selection-agnostic intervals over all rectangles of at least a minimum size, without splitting. It constrains our design: its guarantees do not cover irregular masks. We keep arbitrary candidate masks and instead pay through $\eta$.

WinCLIP, DenseCLIP, and FOV-Seg supply the region proposers. We test the outputs of these methods rather than competing with them. The sample-splitting literature documents the tax we are trying to avoid. For example, KSDAGG shows that split-based tests on high-dimensional data fail to gain power as data grow, while aggregation over the full sample does not.

Research Program

Candidate masks. For an image and prompt, we build a finite candidate set of masks: connected components of thresholded patch-score maps at several thresholds, plus WinCLIP-style multi-scale windows. The set is fixed by a rule decided before seeing scores, but its members depend on the image. The stability bound is unaffected by this, since it needs only the bounded score range.

Contrast statistic. Each patch's score is first adjusted for the text-side attention sink by subtracting the score against a start-token-only "empty prompt" embedding. This is a direct response to the similarity reversal reported in CLIP Tricks You. Each candidate's contrast is then the difference between the mean adjusted score inside the mask and the mean outside it, clipped to a pre-declared range.

Selection and test. One mask is drawn with the exponential mechanism at temperature $\varepsilon$. We then compute a standardized contrast for the drawn mask, with its variance estimated by spatial block resampling to account for correlation between neighbouring patches. This statistic is compared against the normal quantile at level $\alpha e^{-\eta}$. The guarantee is

$$\Pr_{H_0}\big(\text{reject on selected } \hat M\big) \le e^{\eta}\,\alpha_{\text{base}}, \qquad \eta = \varepsilon B,$$

where $\alpha_{\text{base}}$ is the level of the fixed-region test and $B$ is the width of the clipped score range. The fixed-region test is asymptotically valid as the patch count grows, assuming the concept-absent score field is weakly dependent. That assumption is checked directly rather than taken on faith.

Measuring the global component's effect. The analytical component quantifies how the shared component sets power. For each backbone (ViT-B/16 and ViT-L/14, where ClearCLIP reports worse dense maps at larger scale), we measure the ratio of the typical inside-versus-outside contrast to $B$, with and without surgery. We then predict power from that ratio before running the tests. If the prediction holds, the ClearCLIP "scaling paradox" becomes a quantitative statement about stability cost. A secondary, practical output is a reference implementation that adds roughly one forward pass of overhead per prompt.

Baselines

The naive reference point is unadjusted testing. We take the top-scoring mask deterministically and run the same block-variance z-test at level $\alpha$ on the same patches. This is the double-dipping procedure that the image-segmentation selective-inference papers show reaching near-100% false alarms on null images. We expect the same here, and it calibrates how much correction is needed.

The strongest valid competitor is reference-set calibration, which is plain data splitting at the image level. We hold out $n$ images known to lack the concept, run the deterministic top-mask selector on each, and reject on a test image if its selected contrast exceeds the conformal $(1-\alpha)$ quantile of the $n$ reference values. This is exactly valid under exchangeability and needs no spatial-dependence assumption, which is why it is the method to beat. It is run on the same backbone, surgery, prompts, and candidate rule, with $n$ swept from 5 to 200. Beating it means matching its false-alarm rate while achieving higher power at every $n$ below some crossover, and characterizing where that crossover lies.

The second competitor is an intra-image checkerboard split. Masks are proposed from scores on odd patches and tested on even patches, using the same statistic and variance estimator.

The third is Bonferroni over the finite candidate set. Every candidate is tested at $\alpha/K$, where $K$ is the number of candidates, and we take the most significant.

Where compute allows, we also adapt the attention-map selective-inference test of Statistical Test for Attention Maps in Vision Transformers to the vision tower of ViT-B/16. Note that it targets pixel intensity rather than prompt match. That paper reports Type I error controlled at about 5% over 10,000 null images. No reported result was found on Lacuna for its power or runtime on vision-language dense prompts, so we measure both ourselves.

Two ablations isolate the key ingredients. The first runs the full method with and without global-component removal, comparing no surgery, ClearCLIP surgery, and simple subtraction of the projection onto the global token. The second runs it with and without the text-sink correction. A third diagnostic ablation randomizes the selection but omits the $e^{-\eta}$ adjustment, which shows how much of the validity comes from the correction itself.

All methods share the same candidate generator, prompts, and $\alpha = 0.05$. The temperature $\varepsilon$ is tuned only on a separate pilot split that is excluded from all reported numbers.

Why not just…?

Why not just calibrate on held-out concept-free images? This is the real threat. When plentiful, matched reference images exist, it is simpler and exactly valid. Our case rests on two regimes where it struggles. The first is small $n$: with 19 references, the smallest attainable conformal level is 0.05. The second is distribution shift between reference and test images, which breaks exchangeability while a within-image null is unaffected. The $n$-sweep and a cross-dataset shift experiment (calibrating on COCOStuff and testing on ADE20K) settle this. If reference calibration wins even at $n = 20$ under shift, the proposal's value reduces to the diagnostic finding about the global component, and we will report it that way.

Why not just split the image like a checkerboard? Halving the patches is cheap, but adjacent patches are strongly correlated through attention and overlapping receptive fields. The two halves are therefore not independent, and the split's validity is no better grounded than ours, while it discards half the signal. We will measure its false-alarm rate on concept-absent prompts. We expect inflation from the correlation and lower power from the halved sample.

Why not just apply Bonferroni over the candidate masks? With a modest candidate set, this is valid and simple. Its cost grows roughly as $\sqrt{2\log K}$ in the critical value, whereas the stability penalty depends on $B$ and $\varepsilon$ but not on $K$. The prediction is that Bonferroni wins for small $K$ and loses for large multi-threshold, multi-scale candidate sets. We report power as a function of $K$ to locate the crossover. A crossover beyond realistic $K$ would count against the method.

Why not just trust tuned confidence scores, as WinCLIP does? Harmonic prompt ensembles reach 85.1% zero-shot pixel-level segmentation AUROC on MVTec-AD (WinCLIP). However, AUROC measures ranking, not false-alarm control. A threshold tuned on one category does not carry a guarantee to the next. We report per-category false-alarm rates for thresholds transferred across categories to make that gap concrete.

Proposed Experiments

Concept-absent and concept-present prompts. On COCOStuff, ADE20K, and PASCAL VOC we use ground-truth labels to form two kinds of prompt per image: absent classes, which give the false-alarm rate, and present classes, which give power and the IoU between the selected mask and the true object. As a sanity anchor, ClearCLIP reports 23.9 mIoU on COCOStuff with ViT-B/16 against 4.4 for vanilla CLIP. We first reproduce the dense maps before testing them.

Industrial anomalies. On MVTec-AD and VisA, defect-free test images give false-alarm rates for prompts such as "a scratch," and defective images give power.

Medical scans. On the Brain Tumor MRI scans used by the selective-inference papers, we report our semantic test next to their intensity tests. This shows that the two hypotheses differ in practice, not as a head-to-head on a shared metric.

Decisive checks. Four results decide whether the idea works.

The first is validity. The empirical false-alarm rate on concept-absent prompts must stay at or below 5%, with a binomial confidence interval, across backbones and datasets. A prerequisite is a direct check on concept-absent images that block-variance z-scores for fixed random masks are approximately standard normal.

The second is the global-component prediction. Removing the shared component should raise power at matched validity, and more for ViT-L/14 than for ViT-B/16. The measured contrast-to-$B$ ratio should predict the size of that gain.

The third is the data-tax comparison. We locate the reference-set size $n$ at which held-out calibration overtakes the split-free test, both in-distribution and under shift.

The fourth is that the sink-corrected statistic should beat the raw similarity statistic in power on RefCOCO-style prompts. Failure there would indicate that the reported similarity reversal does not transfer to the dense setting.