Trust-Region Control for Iterative Mask Refinement

Many segmentation systems now improve a mask by running it through the network several times. Each pass takes the image and the current mask and produces a corrected mask. How many passes to run, and how much each pass may change, is decided by hand-set constants. GenSAM, for example, always runs six coarse-to-fine cycles, and SAM-REF accepts or rejects each update with a single error-ratio threshold. The result is that the right setting for thin wires is wrong for camouflaged animals. Too few passes leave errors in place, and too many let the mask drift and erode fine boundaries. Numerical optimization faced the same problem decades ago and solved it with trust regions. A trust-region method compares how much a step was predicted to help with how much it actually helped. It then grows or shrinks the allowed step size, and it can shrink a step partway rather than only accepting or rejecting it.

We propose a training-free controller that wraps any existing refiner and applies this logic to masks. At each pass, the controller compares the improvement the refiner implicitly predicts with the improvement an independent quality estimator measures. That ratio sets how many pixels the next pass may flip and when to stop. The controller borrows one idea from network coding as a secondary hypothesis, and it is the part we are least sure of. Noncoherent network coding encodes information in the subspace a set of packets spans, so that unknown linear mixing by the network does not destroy it. By analogy, we measure "how much the mask changed" as a distance between feature subspaces rather than in raw pixels or raw feature coordinates. Our claim is that one setting of the controller's own constants then transfers across backbones and datasets, where fixed iteration counts must be retuned for each.

Core Claim

The claim has three parts. First, a refinement loop controlled by an actual-versus-predicted improvement ratio, with partial (truncated) updates, matches or beats the best per-dataset-tuned fixed iteration count while using fewer passes on average. Second, it avoids the late-iteration degradation that ungated loops show. Third, its meta-parameters, chosen once on one dataset, stay near-optimal on the others.

The closest existing work is SAM-REF's Dynamic Selector. Its SAM-REF gate computes an error ratio across successive iterations and makes a binary keep-or-discard decision. Our controller differs in three ways. It keeps a step size that persists across passes, it can accept part of an update, and it decides when to stop. DynaMask adapts mask resolution to instance complexity, but it does so in a single forward decision, with no feedback trajectory. GenSAM's progressive mask generation GenSAM motivates the problem directly, because its iteration count is fixed at six whatever the image.

Research Program

The controller assumes three components from the refiner. The first is a refiner $R$ that maps an image and mask $m_k$ to per-pixel foreground probabilities $p_{k+1}$. RITM, SAM-REF's refiners, SAMRefiner, and GenSAM's progressive generation all fit this description. The second is a quality estimator $Q$ that needs no ground truth. Our default is SAM's IoU-prediction head, and we test a PseudoClick-style error decoder PseudoClick as an alternative. The third is a radius $\Delta_k$, which is the fraction of currently disagreeing pixels the controller may flip at pass $k$.

Each pass works as follows. The candidate update flips only the $\Delta_k$ fraction of disagreeing pixels on which the refiner is most confident. This mirrors the truncation step that A Trust-Region Method For Nonsmooth Nonconvex Optimization uses to keep sparse penalties from stalling steps. The predicted gain $g_k$ is the refiner's own confidence margin summed over the flipped pixels and normalized by mask area. The controller then forms the ratio

$$\rho_k = \frac{Q(m_{k+1}) - Q(m_k)}{g_k},$$

where $Q(\cdot)$ is the estimated mask quality. Following the standard trust-region update in that same paper, the controller acts on $\rho_k$ in three ways. If $\rho_k < \eta_1$, it rejects the step and shrinks the radius, $\Delta_{k+1} = r_1\Delta_k$. If $\rho_k > \eta_2$, it accepts the step and expands the radius, $\Delta_{k+1} = \min(r_2\Delta_k, 1)$. In between, it accepts the step and keeps the radius. The loop stops when $\Delta_k$ falls below a floor or the pass budget runs out.

The controller has four meta-parameters: $\eta_1$, $\eta_2$, $r_1$, and $r_2$. Milzarek et al. note that such constants can make the controller itself oscillate. We therefore treat the sensitivity of these four constants as a primary measurement, not an afterthought.

Three things this design does not claim should be stated plainly. Trust-region convergence proofs do not carry over to a neural refiner. All we can guarantee is that $Q$ never decreases on an accepted step, which is monotone in the estimator, not in true IoU. How well $Q$ tracks true IoU is therefore measured explicitly. That calibration is the main risk to the whole method.

The subspace hypothesis supplies the stopping and drift signal. At each pass we take the backbone features inside the mask, form a $c \times n$ matrix of $n$ foreground feature vectors with $c$ channels, and keep its top-$d$ left singular subspace $U_k$. The change between passes is the Grassmann distance

$$\delta_k = \Big(\textstyle\sum_{i=1}^{d} \theta_i^2\Big)^{1/2},$$

where $\theta_i$ are the principal angles between $U_k$ and $U_{k+1}$. This distance does not change under any orthogonal mixing of feature channels, which is the same invariance noncoherent network coding exploits (On the Capacity of Noncoherent Network Coding; On the capacity of multisource non-coherent network coding). The analogy is imperfect. Exact invariance under any invertible mixing holds only when the matrix spans its subspace exactly, as in the coding setting. Here we truncate to the top $d$ directions, so the invariance is only approximate for non-orthogonal transforms. We also expect $\delta_k$ to stay roughly scale-free across masks of different sizes, because it ignores how many pixels the mask contains. We use $\delta_k$ in two ways: as a stopping test ($\delta_k$ below a threshold means the semantic content of the mask has stopped moving), and as a drift alarm (a large $\delta_k$ alongside a small pixel change suggests the mask has jumped to a different object part). We do not adopt rank-metric lifting or syndrome decoding. Section 4.4 of the research file raises them as an option, but nothing in the file suggests mask errors have the algebraic low-rank structure those decoders assume, so we leave them aside.

Baselines

The naive reference point is a single refinement pass, together with the backbone's original output. This shows how much iteration helps at all.

The second baseline is a fixed iteration count tuned per dataset on held-out data. This is the oracle-tuned static schedule. It is the competitor a skeptic would favor, and the controller must at least match it while using one meta-parameter setting for every dataset.

The strongest existing method is the SAM-REF Dynamic Selector SAM-REF. The paper reports 90.75% 5-mIoU on the HQSeg-44K validation set, against 88.0% for SAM, and NoC90 of 1.43 clicks on Berkeley, against 2.09 for SAM. For automatic correction, SAMRefiner SAMRefiner reports 87.1% IoU on DAVIS-585, against 81.9% for CascadePSP. On camouflage, GenSAM GenSAM reports MAE 0.067 on COD10K with its fixed six iterations.

To keep comparisons fair, every controller variant wraps the identical frozen refiner and estimator. Each variant gets the same maximum pass budget, and wall-clock time is measured on the same GPU. Each baseline's own thresholds are tuned with the same validation budget we use for our meta-parameters: an equal number of configurations, all tuned on HQSeg-44K only.

Two ablations isolate the key ingredients. The first is the controller with binary accept/reject only, meaning no truncation and a fixed radius. It separates "graded step size" from "gating." The second replaces the Grassmann stopping signal with pixel IoU between successive masks. It separates the subspace idea from plain convergence detection.

Beating the field means two things. On quality, the controller must match or beat SAM-REF's gate and the oracle fixed count on 5-mIoU, mBIoU, and MAE within seed noise. On cost, it must use fewer mean passes, and its untuned transfer must lose less than the static schedule loses when that schedule is transferred untuned.

Why not just…?

Why not just tune the iteration count per dataset? It may work as well within a dataset, and the proposed experiments will report exactly that. The case against it is twofold. It must be retuned for every new domain. It also spends the same number of passes on easy and hard instances, a waste DynaMask showed matters when it adapted resolution per instance. If the transferred fixed count loses no more than our controller under cross-dataset transfer, the main claim fails.

Why not just stop when the IoU head's score stops rising? This simple form of quality-based early stopping is a serious threat. It shares our estimator $Q$, and it amounts to our controller without the radius and without partial steps. We include it as a baseline. If it matches the full controller, the graded step size adds nothing and we would report that. The argument for the step size is that over-refinement typically damages a few fine structures while improving the bulk of the mask. Whole-mask accept/reject cannot separate those two effects, whereas truncated flips can. The DIS boundary metric is where this difference should show.

Why not just learn a halting policy end-to-end? A learned stopping network needs training trajectories on each refiner, and it introduces exactly the dataset-dependence we are trying to remove. The analogy is Borgelt's finding that tuned acceleration constants in iterative clustering diverge on other data. We will train a small halting head on HQSeg-44K trajectories and test its transfer alongside our controller. A learned policy that transfers equally well would undercut the training-free argument.

Why bother with subspaces instead of pixel change? Pixel change may be enough. The ablation decides it, and the subspace signal is dropped if it shows no gain in cross-backbone threshold transfer from SAM ViT-H to HQ-SAM.

Proposed Experiments

We wrap three refiners: SAM-REF's refiners, SAMRefiner on noisy pseudo-labels, and GenSAM's progressive generation. They are evaluated on HQSeg-44K (5-mIoU), DIS (mBIoU, where HQ-SAM HQ-SAM reports 70.4), DAVIS-585 (IoU), COD10K, CAMO, and CHAMELEON (MAE), and Berkeley and GrabCut (clicks to 90% IoU). For GrabCut, RITM RITM reports 1.42 to 1.56 clicks. For SAM-REF on GrabCut and for LVIS, no reported result was found on Lacuna.

The meta-parameters are tuned once on HQSeg-44K and then frozen. We report a sensitivity heat map over $(\eta_1, \eta_2)$ on every dataset, quality-versus-pass-count curves, the rank correlation between $Q$ and true IoU across passes, and the fraction of instances whose true IoU falls on an accepted step.

Three checks decide the outcome. On COD10K, the transferred controller must match or beat GenSAM's six-iteration MAE with fewer mean iterations. On DIS, truncated steps must beat binary gating on mBIoU. And the controller's untuned loss on each held-out dataset must be smaller than that of the transferred fixed count.