Segmentation models that refine their own masks over several passes have a known failure: each new pass can damage parts of the mask that were already correct. A model that should be sharpening the edge of a bicycle spoke may instead shift the whole wheel, or erase a thin cable it found in an earlier pass. We propose a simple constraint that targets this failure. Each refinement step may only change the mask at a scale it is entitled to change. We represent the mask in a wavelet basis, which splits an image into a coarse approximation plus detail coefficients at successively finer scales and in horizontal, vertical, and diagonal orientations. The first iterations may edit only coarse coefficients. Later iterations may edit only fine coefficients, and in interactive use, only near the user's latest click. Fine-scale corrections then cannot move the object's overall layout, and coarse corrections cannot produce spurious speckle.
The constraint is only as good as the geometry it runs on. The finest detail bands of a thin object live at the pixel scale. If a 1:4 image of a wire fence is squashed into a 1024×1024 square, wires that run across the compressed axis fall below one pixel and alias away before any refinement starts. The refiner therefore runs on windows that keep the object's native aspect ratio and resolution. Aspect ratio here is a constraint the method must respect, and it defines the regime where we expect the biggest gains: thin objects that are long in one direction.
Core Claim
Restricting each refinement step's update to a fixed set of wavelet bands of the mask should do two things on thin structures. It should reduce how often refinement breaks pixels that were already correct, and it should raise final boundary accuracy. It should do both better than an unrestricted refiner of identical capacity, and better than a heuristic accept-or-reject gate. The benefit should be largest when the refiner works at native aspect ratio, because that is when the fine bands actually contain the structure.
Let $m_t$ be the soft mask after step $t$. Let $W$ be a $J$-level discrete wavelet transform, and let $\Delta_t$ be the coefficient update predicted by the refiner network. The update rule is
$$m_{t+1} = W^{-1}\big(W m_t + P_{B_t}\,\Delta_t\big),$$
where $P_{B_t}$ zeroes every coefficient outside the allowed band set $B_t$. For $t=1$ the set $B_t$ holds the coarsest levels, and later steps move toward the finest levels. In interactive mode, the fine-level entries of $B_t$ are also restricted spatially to a neighbourhood of the newest click. This structural constraint plays the role that SAM-REF fills with its Dynamic Selector, a gate that rejects updates whose estimated error rises (SAM-REF).
Positioning
The closest work on iterative refinement feeds the previous mask back as an extra input channel and lets the network rewrite every pixel. This covers the hypercolumn loop of Iterative Instance Segmentation, which gained 9.3 AP^r points at 70% overlap over three steps. It also covers PRISM and the error-map-driven PseudoClick. These papers motivate the proposal, because they show feedback helps. They also constrain it: the file records drift and multi-round regressions as open problems, and nothing in these methods bounds what a step may change. SAM-REF is the strongest refinement-loop competitor. It adds a global refiner on a downsampled mask and a local refiner on square high-resolution crops around clicks, all in the spatial domain on top of SAM's fixed square encoder. We keep its two-scale intuition but enforce scale separation in the output parameterization rather than through an after-the-fact gate. We also replace square crops with native-aspect windows.
Frequency decomposition enters segmentation in FP-DIS (Dichotomous Image Segmentation with Frequency Priors), which applies cosine-transform band filters to image features and reports a 4.4% weighted F-measure gain over IS-Net on DIS5K. It also enters through EW-ViT (Frequency-Domain Refinement of Vision Transformers), which uses wavelet downsampling to keep boundaries stable under noise. Both decompose the image in a single forward pass on square-resized inputs. We decompose the mask update across iterations, and we borrow their image-band features only as conditioning for the fine-band updates. That conditioning is the one place the frequency-decoupled restoration idea is used directly: separating smooth from detailed content keeps a correction for one from corrupting the other (Lugmayr et al.).
On geometry, NaViT and LLaVA-UHD show that preserving native aspect ratio helps recognition. TOS-Net shows that a native-resolution edge stream matters for thin objects. We reuse NaViT-style packing only so that variable-shape windows can be batched. We do not claim a new aspect-ratio method.
Research Program
The coarse stage is a frozen HQ-SAM, used unchanged so that every comparison shares the same starting mask. The refiner is a lightweight convolutional network. It reads an object-aligned window cropped at native resolution and native aspect ratio, tiled with overlap when the window is too large. Its inputs are the window's image, the wavelet bands of that image, the wavelet coefficients of $m_t$, and click maps. It outputs $\Delta_t$, and the loop runs three to four steps.
We train on HQSeg-44K with simulated clicks placed in the largest error regions, in the style of PseudoClick. The loss is a per-step boundary-aware segmentation loss on every $m_t$. The band schedule is fixed: coarse levels at step 1, then each successive step allowed one finer level. A learned schedule is a later variant, and only if the fixed one already works.
The analysis has three parts. First, we measure the regression rate, the fraction of pixels correct at step $t$ that become wrong at step $t+1$, at every step. Second, we measure orientation-resolved boundary error, computed separately for boundary segments along the long and short axes of the object. This tests whether squashing specifically destroys detail across the compressed axis. Third, we measure the effect of image degradation by adding Gaussian noise up to σ=0.2, the range EW-ViT used, and blur. This checks whether fine-band mask updates track real edges rather than noise.
Baselines
The naive reference point is the frozen HQ-SAM mask with no refinement. HQ-SAM itself reports DIS boundary IoU (mBIoU) rising from 52.8 for base SAM to 70.4 (Segment Anything in High Quality). Our refiner must first clear this starting point on the same splits.
The strongest existing competitor is SAM-REF, the closest published method that both refines over multiple rounds and recovers high-resolution detail. It reports 5-mIoU of 90.75% on HQSeg-44K validation against 88.0% for SAM, and Berkeley NoC90 of 1.43 against 2.09. NoC90 is the number of clicks needed to reach 90% IoU. We found no reported SAM-REF result on DIS5K or ThinObject-5K on Lacuna, so we will rerun it. It will see the same HQ-SAM coarse masks, the same HQSeg-44K training data, the same click simulator, and the same GPU-hour and hyperparameter-search budget as our refiner. Beating it means a higher boundary IoU on the elongated subsets, defined below, at equal or fewer clicks and comparable latency (about 0.5 s per click as reported). We also report TOS-Net's 65.1% IoU_thin at four clicks and IS-Net's 0.753 weighted F-measure on DIS-TE4 (Highly Accurate Dichotomous Image Segmentation) as published reference points.
The ablations matter most. The key-ingredient ablation is the identical refiner and inputs with $P_{B_t}$ removed, so every step may edit every coefficient. A second ablation keeps the unrestricted refiner but adds a SAM-REF-style accept-or-reject gate. A third swaps the wavelet basis for a spatial Laplacian pyramid, a stack of blurred-and-differenced images that is a standard spatial multi-scale decomposition, with the same band schedule. A fourth runs the full method on square crops, and a fifth on squashed square windows instead of native-aspect windows.
Why not just…?
Why not just feed back a blurred or soft mask instead of restricting bands? Blurring the fed-back mask removes the sharp step edges at the input. It does not limit what the output may change, so it cannot prevent a late step from shifting the coarse layout. That makes it a weaker form of our idea. We include it as a variant of the unrestricted ablation. If it matches band restriction on regression rate, our structural claim fails.
Why not just use a Laplacian pyramid or other spatial multi-scale refiner? This is a real threat. The point of our claim is band-restricted updates, not wavelets as such, and a pyramid schedule might work equally well. The pyramid ablation settles it. If it ties, we report the result as a finding about scale-scheduled updates in any basis. The remaining wavelet-specific claim would then rest only on its separate horizontal and vertical detail bands helping on elongated objects, which the orientation-resolved error measures directly.
Why not just pad to square, or use SAM-REF's square crops? Padding avoids distortion, but on a 1:4 window most of the square is empty, so for a fixed input size the object gets far fewer pixels. NaViT documents exactly this waste for padding (Patch n' Pack). Square crops around a click cut an elongated object away from its continuation. Both are run as baselines at matched pixel budgets. If padded native-aspect input matches packed windows, we adopt padding and drop the packing machinery.
Why not just a bigger one-shot decoder? HQ-SAM's own gap over SAM came from targeted low-level detail, not scale. Iterative feedback beat added feedforward capacity in Iterative Instance Segmentation. We test this directly by giving a single-pass decoder the refiner's total compute across all steps.
Proposed Experiments
We evaluate on DIS5K (DIS-TE1 to TE4), ThinObject-5K, the HQSeg-44K validation split, and Berkeley and DAVIS for interactive click counts. Results are also reported on an elongated subset of each set, defined as objects whose bounding box exceeds 3:1. The metrics are mBIoU, weighted F-measure, IoU_thin, NoC85 and NoC90, per-step regression rate, orientation-resolved boundary error, and latency.
Three checks decide the proposal. First, the band-restricted refiner must show lower regression rate and higher final mBIoU than the unrestricted ablation at steps three and beyond, on the elongated subsets. If it does not, the core claim is false. Second, there must be an interaction: native-aspect windows should help the band-restricted refiner more than they help the unrestricted one. Squashing should hurt mainly the short-axis boundary error, the error along boundaries spanning the compressed axis where thin structures alias away. Third, under added noise, conditioning fine-band updates on image detail bands should keep precision on thin parts from collapsing relative to the spatial-only SAM-REF rerun. If only the first check holds, the contribution narrows to a regression-free refinement rule.
References
- SAM-REF: Introducing Image-Prompt Synergy during Interaction for Detail Enhancement in the Segment Anything Model
- Iterative Instance Segmentation
- PRISM: A Promptable and Robust Interactive Segmentation Model with Visual Prompts
- PseudoClick: Interactive Image Segmentation with Click Imitation
- Dichotomous Image Segmentation with Frequency Priors
- Frequency-Domain Refinement of Vision Transformers for Robust Medical Image Segmentation Under Degradation
- NTIRE 2020 Challenge on Real-World Image Super-Resolution: Methods and Results
- Patch n’ Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution
- LLaVA-UHD: An LMM Perceiving Any Aspect Ratio and High-Resolution Images
- Deep Interactive Thin Object Selection
- Segment Anything in High Quality
- Highly Accurate Dichotomous Image Segmentation