Top-Down Feedback That Moves Where Convolutions Look

Vision networks look at an image through small, fixed square windows, and they process it in a single bottom-up pass. Deformable convolution relaxes the first constraint. It is a layer that learns, for each pixel, small shifts that move its sampling points off the square grid, so a kernel can bend along a vessel or stretch over an object. Recurrent feedback relaxes the second constraint by letting a network's current high-level guess revisit and reinterpret earlier features. The two fixes have not been joined in the place where they would matter most. Deformable offsets are always computed from local features, blind to what the network currently thinks the object is. Feedback, in turn, only reweights or gates features that were already sampled on a rigid grid. We propose a recurrent network in which the top-down hypothesis, the current coarse prediction of where the object and its boundary are, directly adds a correction to the sampling offsets of early deformable layers. The network is unrolled over a few iterations with shared weights and stops early on easy images.

The case for this rests on a simple asymmetry. Gating can suppress a background feature that was wrongly sampled, but it cannot fetch an object feature that was never sampled. On camouflaged objects and on thin structures broken by low contrast, the dominant failure is geometric. Deformable sampling points drift into the background, a problem documented directly by Deformable ConvNets V2. A feedback signal that moves the sampling points should therefore help most on exactly the hard images where feedforward networks fail.

Core Claim

On tasks where local evidence is ambiguous, steering deformable offsets with a recurrent top-down hypothesis beats two cheaper alternatives at matched compute. The first is using the same feedback to gate features. The second is feeding the hypothesis to an ordinary bottom-up offset predictor. The gains should concentrate on hard images: camouflaged instances, and vessel segments with occluded gaps. They should come with a measurable shift of sampling points onto the true object.

Concretely, at deformable layer $l$ and iteration $t$, the offset for kernel tap $n$ is

$$\Delta p_n^{(l,t)} = g_l\big(x^{(l,t)}\big) + h_l\big(u^{(t-1)}\big),$$

where $g_l$ is the usual bottom-up offset head applied to the layer's features $x^{(l,t)}$. The state $u^{(t-1)}$ is the top-down hypothesis from the previous iteration: the decoder's coarse mask together with its signed distance and boundary-tangent fields. The function $h_l$ is a small learned head that maps that hypothesis to a displacement field. The modulation scalar of DCNv2 is kept. The correction $h_l$ is bounded with a tanh, in the style of offset-decoupled deformable convolution, so feedback cannot throw sampling points arbitrarily far.

Where This Sits

Three lines of work motivate the proposal.

The recurrence literature shows that one bottom-up pass is insufficient on ambiguous images:

  • Kar et al. find that primates need roughly 30 ms of extra, recurrent-dependent processing on "challenge images," and that feedforward activations fail to predict late responses in inferior temporal cortex.
  • Spoerer et al. 2017 show that recurrent convolutional networks with lateral and top-down connections beat parameter-matched feedforward controls on occluded digits.
  • Spoerer et al. 2020 show that shared-weight recurrence can stop early on easy images by thresholding prediction entropy. We reuse that halting rule.

None of these works changes where kernels sample. Their feedback acts on fixed square grids.

The deformable literature supplies the operator and its known failure:

  • DCNv1 introduces learned fractional offsets sampled with bilinear interpolation.
  • DCNv2 documents sampling points bleeding into the background.
  • DCNv4 makes the operator fast by attacking memory-access cost. This is what keeps several recurrent iterations affordable.
  • Dynamic Snake Convolution chains the sampling points so they follow tubular structures, and adds a topology loss.

All of these compute offsets feedforward from local features. The snake constraint in particular cannot recover a segment it has already lost to low contrast.

The works that come closest to combining feedback and adaptive sampling constrain the design most directly:

  • ESCNet pairs deformable convolutions with a hierarchical feedback cascade for camouflaged objects. Its feedback refines features by multiplicative attention, and its offsets come purely from local texture.
  • GRAFNet projects high-level hypotheses back to early layers by predictive coding, but uses fixed-orientation steerable filters rather than free offsets.
  • Look and Think Twice runs 10–50 gradient-ascent iterations per image to update activation gates. It leaves the grids rigid and is costly at inference.

Our proposal differs from all three in one respect: what feedback writes to is the sampling geometry itself.

Research Program

We build one module, the feedback-steered deformable layer, and insert it into the first two deformable stages of two established pipelines.

Pipelines. For camouflaged object detection we use a PVTv2-backbone segmentation network in the style of ESCNet. For retinal vessels we use a U-Net in the style of DSCNet.

Recurrent loop. The decoder output at iteration $t-1$ is converted into mask, signed-distance and tangent maps, then downsampled to each deformable layer's resolution to form $u^{(t-1)}$. Weights are shared across iterations. We train for $T=3$ unrolled steps with deep supervision at each step, and at test time apply entropy-based halting with $T$ up to 5.

Initialisation. The first iteration has no hypothesis, so $h_l$ receives zeros. The model therefore starts as an ordinary deformable network.

Analysis. Beyond accuracy, we measure the on-object sampling ratio: the fraction of deformable sampling points that fall inside the ground-truth mask, or within a vessel's width. We track it across iterations and split it into easy and hard strata. Hard images are defined in advance as the lowest-confidence tercile under the feedforward baseline. For vessels, a second hard stratum consists of test images with synthetic occluding blotches placed over known vessel segments.

Baselines

Naive reference. The pipeline with standard DCNv2 layers, run once. It shows what adaptive sampling without feedback achieves, and fixes the floor every other variant must clear.

Strongest published methods. On CAMO, the best number in our file is ESCNet at structure-measure 0.892 and mean absolute error 0.028 (ESCNet). On COD10K, no reported ESCNet result was found on Lacuna. The only file figure is SINet at structure-measure 0.771. We will therefore retrain ESCNet under our protocol rather than quote it. On DRIVE, DSCNet reports Dice 82.06% against 80.73% for U-Net, and DUNet reports AUC 0.9856. Beating DSCNet means exceeding its Dice by more than seed variance, with a clearly larger gain on the occluded-gap stratum and on topology error.

Compute-matched feedforward control. A deformable network deepened or widened to match the recurrent model's multiply-add count at $T=3$. This follows the depth- and kernel-matched controls of Spoerer et al. 2020.

Ablations of the key ingredient. Each of the following keeps the architecture, iterations, parameters and compute fixed, and changes only where feedback enters:

  • Gate ablation. Feedback multiplies features, as in ESCNet and GRAFNet.
  • Input ablation. The hypothesis maps are concatenated to the inputs of the bottom-up offset head $g_l$.
  • Grid ablation. Feedback enters rigid convolutions, in the style of the BLT recurrent network.

Fairness protocol. All models share data splits, augmentation, training epochs and a fixed budget of twelve hyperparameter trials each. Results are reported over three seeds.

What counts as success. The proposal wins if offset steering beats every ablation on the hard strata by a margin larger than the spread across seeds, while roughly matching them on easy images.

Why not just…?

Why not just make the deformable network deeper at the same compute? Parameter- and depth-matched feedforward controls lost clearly to recurrence on occluded digits: 29.6% versus 9.3% error on heavy debris in Spoerer et al. 2017. They also lost on ImageNet: 62.68% versus 64.37% top-1 in Spoerer et al. 2020. That evidence comes from rigid grids, so our compute-matched control tests whether it carries over to deformable ones. If the deeper network matches ours on the hard strata, the recurrence claim fails.

Why not just feed the coarse mask into the ordinary offset head? This is the real threat. It lets the bottom-up predictor see the hypothesis, and it is nearly free to implement. The distinction we are betting on is the following. A separate, bounded correction head specialises in moving points toward the hypothesised boundary. Concatenation instead forces one head to mix texture and hypothesis, and it may learn to ignore the latter. We do not argue this; the input ablation settles it. If input concatenation performs as well, we will report that the benefit comes from recurrent access to the hypothesis rather than from a dedicated steering pathway, and simplify the method accordingly.

Why not just use global attention to compute the offsets? Global context predicted in one pass, as in DAS or attention backbones, can widen what the offset head sees. It still cannot revise sampling after seeing its own mistakes. Full attention is also quadratic in pixel count, which CCNet quantifies as roughly 11 times the memory of its factorised alternative. The camouflage pipeline already uses a PVTv2 transformer backbone, so attention-derived context is present in every variant. Any gain we show is therefore on top of global attention.

Why not just use feedback on features, which ESCNet and GRAFNet already do? Gating cannot bring in evidence from locations never sampled. The gate ablation isolates exactly this, and the on-object sampling ratio shows whether the mechanism works as claimed.

Decisive Checks

We will run four tests:

  1. CAMO and COD10K. Structure-measure, mean absolute error, weighted F-measure and E-measure, reported overall and on the hard tercile.
  2. DRIVE and CHASE_DB1. Dice, AUC, and a topology error based on persistent homology, following DSCNet, including the synthetic-occlusion stratum.
  3. Digit Debris sanity test. The heavy-debris task of Spoerer et al. 2017, where BLT's 9.3% error is the reference. This checks whether offset steering adds anything to recurrence alone on a controlled occlusion task.
  4. Latency. Wall-clock time per image using DCNv4-style kernels, both with and without halting.

The idea is confirmed only if all of the following hold together:

  • Offset steering beats the gate, input and grid ablations on the hard strata.
  • Its on-object sampling ratio rises across iterations while the ablations' ratios stay flat.
  • Accuracy on the hard strata improves with more iterations, and entropy halting spends extra iterations on precisely those images.

If gains appear uniformly across easy and hard images, or if the sampling ratio does not move, the benefit is ordinary extra capacity and not top-down geometric correction.