A common way to get segmentation training data without human labeling is to ask a text-to-image diffusion model (a generator that turns random noise into an image over many denoising steps) to draw "a photo of a dog." The pipeline then reads off where the model was "looking" when it processed the word dog. That spatial map, the cross-attention map, becomes a rough mask. The maps are fuzzy, so current pipelines finish every image, clean the masks with extra neural networks, and then throw most of the output away. DiffuMask discards the bottom 70% of its samples, so about 70% of its diffusion compute ends up in the bin. This proposal reuses a decades-old idea from classical vision, maximally stable extremal regions (MSER). MSER means sweeping a threshold over a map and keeping the blobs whose shape barely changes across a wide range of thresholds. We ask whether that stability, measured across thresholds and across denoising steps, can do two jobs at once. It would extract the mask, and it would predict partway through generation whether the sample is going to be thrown away, so the sampler can stop early.
In two sentences: an object the diffusion model has clearly committed to should show up as an attention blob that is stable both across threshold levels and across successive denoising steps, and that stability can be read cheaply long before the image is finished. If that early stability predicts final mask quality, we can abort doomed samples early and spend compute only on samples we will keep. The target is better segmentation accuracy per GPU-hour, not a better mask cleaner.
Core Claim
The obvious idea at this intersection is "replace AffinityNet with MSER." On its own it is weak. Dataset Diffusion already removes the auxiliary affinity network by sharpening attention with class-prompt appending and self-attention exponentiation. It reports 64.8 mIoU on synth-VOC, 4.2 above DiffuMask, without any MSER. A better post-hoc binarizer would compete with that system on mask quality alone, and the margin is likely small.
The claim we make instead concerns where the compute goes. Let $A_t^c$ be the attention map for class token $c$ at reverse step $t$. Let $R_t(\tau)$ be the connected region of $A_t^c$ above threshold $\tau$ that contains its peak. We score a sample at step $t$ as
$$ s_t^c = \max_{\tau}\left(1 - \frac{|R_t(\tau-\Delta)\setminus R_t(\tau+\Delta)|}{|R_t(\tau)|}\right)\cdot \mathrm{IoU}\big(R_t(\tau^*),\,R_{t+k}(\tau^*)\big), $$
where $\Delta$ is the threshold step, $\tau^*$ is the maximizing threshold, and $k$ is a lag of a few denoising steps. The first factor is the standard MSER stability term: one minus the relative area change of the region as the threshold moves by $\pm\Delta$. The second factor checks that the chosen region is not still drifting across timesteps. We hypothesize that $s_t^c$ at an intermediate step, around 30–40% of the way through sampling, ranks samples by their eventual mask quality well enough to abort the bottom fraction early. The survivors' final masks come from the same maximally stable region at the last step. Neither stability-based binarization nor early termination is new by itself. The new part is using stability as a mid-trajectory rejection criterion.
Prior Work and Its Roles
Three works motivate the claim that stability across thresholds works on noisy neural heatmaps, not just photographs. AnomalyNCD binarizes uncalibrated anomaly heatmaps by stability across threshold sweeps (its MEBin step) and reports a 10.8% F1 gain on MVTec AD within its full system. Small Defect Detection Using CNN Features and Random Forests runs MSER on a random-forest probability map and reaches 84.5% detection at 2.4 false positives per image on aerospace weld X-rays. Threshold Selection as a Function of Region Count Stability picks thresholds where the region count stops changing. All three operate on discriminative maps after the fact. None applies stability to generative attention or uses it to decide whether to keep computing.
What the DAAM shows that cross-attention tracks nouns well (58.8–64.8 mIoU on noun segmentation). It also shows that attention bleeds across related objects and adjectives. That supports our premise that localization is readable from attention, and it constrains us: stability will flag diffuse, bleeding maps, but it cannot tell a crisp wrong object from a crisp right one.
DiffuMask and Dataset Diffusion are the systems we compete with, and they show two existing ways to pay for clean masks. DiffuMask trains an affinity network for adaptive binarization and prunes 70% of samples after full generation. Dataset Diffusion sharpens attention, then pays with an uncertainty-aware loss and self-training loops. Text-side pipelines show the same pattern of filtering overhead. DISCO uses task-specific teacher filters, It's All Relative! uses round-trip filtering, and InstructDET applies CLIP filtering to region expressions. In every case, rejection happens after the full generation cost has been spent.
ES-DDPM is the closest early-stopping work, and it addresses a different problem. It shortens the trajectory of every sample by starting from a GAN or VAE draft. We truncate only the samples we will reject, with no external generator. The two approaches can be combined.
Fast MSER supplies the method. Its component-tree extraction is up to 28× faster than OpenCV's MSER implementation, with about 1 ms sub-tree merging on 10-megapixel images. On 64×64 upsampled attention maps, computing $s_t^c$ at a handful of steps should therefore cost a tiny fraction of a single denoising step. We will measure that rather than assume it.
Research Program
We will build on the Dataset Diffusion generation recipe so that mask sharpening is held fixed. The pipeline uses Stable Diffusion with class-prompt appending, aggregates cross-attention across layers, and propagates it with exponentiated self-attention. We add three components.
The first is a stability-based mask extractor that runs on each class token's map at the final step. It handles the known weakness of MSER on flat, low-contrast maps by selecting $\Delta$ from the region-count-stability curve of Threshold Selection as a Function of Region Count Stability rather than hand-tuning it. Where several class tokens claim a pixel, the token whose region has higher $s^c$ wins.
The second is a trajectory monitor that computes $s_t^c$ every few steps. At a chosen checkpoint step, it aborts samples whose minimum class score falls below a quantile threshold. That quantile is calibrated once, on a small hand-labeled pool.
The third is cost accounting that charges every pipeline for all diffusion steps it runs, including those spent on samples that are later aborted or discarded, plus all auxiliary-network training and inference.
The analysis has three parts. First, we measure how well $s_t$ at each step $t$ predicts final mask IoU against human labels. Second, we trace the accuracy–compute frontier as the abort threshold and checkpoint step vary. Third, we break down failures into cases where stability is high but the class is wrong, the case DAAM leads us to expect.
Baselines
The naive reference keeps every sample and binarizes the final-step attention map with a single global threshold tuned on the labeled pool, with no pruning. It establishes the floor and the cheapest cost point. A second naive variant uses Otsu's method in place of the tuned global threshold.
The strongest existing method is Dataset Diffusion itself, run with the authors' recipe, including its uncertainty-aware loss and self-training. It reports 64.8 mIoU on synth-VOC and 34.2 mIoU on synth-COCO with DeepLabV3. This is the right main competitor because it already removed AffinityNet and beats DiffuMask on the same benchmark. Its cost sits in self-training rather than discard.
We will also reimplement DiffuMask-style AffinityNet binarization plus 70% post-hoc pruning on the same generator. DiffuMask reports 70.6 mIoU on VOC 2012, but with a Swin-B student, so that number is not directly comparable to DeepLabV3 results and we will not treat it as such.
All pipelines start from the same prompts and the same Stable Diffusion checkpoint, and all train the same DeepLabV3 student under identical schedules. Each gets an equal hyperparameter-tuning budget on a shared 500-image labeled synthetic pool. Every pipeline is compared at matched total GPU-hours, measured end to end.
Beating Dataset Diffusion means one of two outcomes. We could reach its mIoU on real VOC/COCO validation with at least about 30% less end-to-end compute. Alternatively, we could exceed its mIoU at equal compute by more than the seed-to-seed spread, measured over three seeds.
Three ablations isolate the key ingredient. The first uses the stability mask without early abort, which separates mask extraction from compute savings. The second aborts early on a cheap non-topological score, the entropy or peak value of the attention map, at the same checkpoint and rate. The third aborts at random at the same rate, which separates "fewer samples" from "better-chosen samples." A fourth variant drops the timestep factor from $s_t$, which tests whether cross-step consistency matters beyond single-step MSER stability.
Why not just…?
Why not just threshold the attention map with Otsu or a fixed value? DiffuMask reports that simple thresholding of raw cross-attention was inadequate, which is why it added AffinityNet. Our setting differs, because Dataset Diffusion's sharpened maps may threshold acceptably. That is exactly why the fixed and Otsu thresholds are run as baselines on sharpened maps. If a single threshold matches our final masks, mask extraction is not our contribution, and the claim rests entirely on early rejection. A plain threshold gives no graded confidence for early abort, though the entropy-abort ablation tests whether some equally cheap score does.
Why not just add a neural cleaner such as an affinity network, a promptable segmenter, or CLIP filtering? Neural cleaners can catch semantic errors that stability cannot. They still act after the full image is generated, so the discarded fraction's sampling cost is already spent; that is the 70% loss in DiffuMask. This alternative is a real threat on final accuracy. We will therefore test a hybrid: stability-based early abort followed by one cheap CLIP class check on survivors. If the CLIP-only variant matches the hybrid at equal compute, the stability signal adds nothing.
Why not just use a faster sampler (fewer steps, or an ES-DDPM-style warm start)? Faster sampling cuts the cost of every sample but does nothing about the fraction that gets discarded. ES-DDPM also needs a separate GAN or VAE. The two savings multiply rather than compete, so we will run our monitor on a reduced-step schedule as well. The genuine risk is that with very short schedules the attention maps never settle, leaving no useful checkpoint. The step-wise correlation analysis will show where that breaks.
Why not just abort on attention entropy? This is the simplest rival to the trajectory monitor, and it may well work comparably. It is run as a primary ablation at a matched abort rate. If entropy ranks samples as well as $s_t$, we will report that the early-abort idea holds but the topological machinery is unnecessary.
Proposed Experiments
We will generate synth-VOC and synth-COCO following Dataset Diffusion's prompt protocol. Students are evaluated on PASCAL VOC 2012 val and COCO 2017 val. For reference, Dataset Diffusion reports that a real-COCO-trained DeepLabV3 reaches 54.9 mIoU. The calibration and analysis pool is 500 synthetic images per dataset, hand-labeled at pixel level.
The first decisive check runs before any student training. On the labeled pool, we measure the Spearman correlation between $s_t$ at each candidate step and final-mask IoU. If correlation at 30–40% of the trajectory is not clearly above the entropy score's, the early-abort premise fails and we stop.
The second check is the main result: accuracy–compute frontiers for every pipeline, with mIoU plotted against end-to-end GPU-hours. The third is the failure breakdown, the fraction of retained samples that are stable but wrong-class, which decides whether a semantic check must remain. As a secondary test of generality, the same monitor will be applied to multi-object prompts, where Dataset Diffusion reports entangled and missing objects. There we check whether per-token stability conflicts flag those failures early.
References
- DiffuMask: Synthesizing Images with Pixel-level Annotations for Semantic Segmentation Using Diffusion Models
- Dataset Diffusion: Diffusion-based Synthetic Data Generation for Pixel-Level Semantic Segmentation
- AnomalyNCD: Towards Novel Anomaly Class Discovery in Industrial Scenarios
- Small Defect Detection Using Convolutional Neural Network Features and Random Forests
- Threshold Selection as a Function of Region Count Stability
- What the DAAM: Interpreting Stable Diffusion Using Cross Attention
- DISCO: Distilling Counterfactuals with Large Language Models
- It's All Relative! - A Synthetic Query Generation Approach for Improving Zero-Shot Relevance Prediction
- InstructDET: Diversifying Referring Object Detection with Generalized Instructions
- Accelerating Diffusion Models via Early Stop of the Diffusion Process
- Fast MSER