Second Looks for Hard Pixels: Weight-Shared Review Passes in High-Resolution Vision Networks

Networks for dense prediction label every pixel, as in segmentation, pose keypoints, or depth. They work best when they keep a sharp, full-detail copy of the image features all the way through. Doing so is expensive, so the small versions of these networks are thin and shallow, and they lose accuracy mostly at object boundaries, thin structures, and small parts. This proposal gives those hard regions extra processing without adding new weights. A small scoring module picks the image windows that look difficult, and those windows go through the same block a second time, reusing its weights. The rest of the image passes through unchanged. Parameters stay essentially fixed, and the added compute scales with the fraction of the image that is hard rather than with the size of the image.

The idea comes from language modeling. DND showed that sending only uncertain tokens through a nested second pass of existing layers improves small language models by up to 2.61% average accuracy, with no extra transformer blocks and about 6% extra compute. Vision networks that maintain high resolution do the opposite: they spend identical depth on sky and on bicycle spokes. If the DND result carries over, lightweight high-resolution backbones could recover part of the gap to heavy ones. In pose estimation that gap is large: Lite-HRNet-30 reaches 70.4 AP against 76.3 AP for HRNet-W48 on COCO keypoints (Lite-HRNet; HRNet).

Core Claim

In lightweight high-resolution networks, the remaining errors are concentrated in a small, predictable set of spatial windows. Giving those windows a second pass through the same weights recovers more accuracy per added FLOP and per added parameter than three cheaper options: making the network deeper or wider everywhere, adding unshared blocks, or applying the second pass uniformly to all windows. The new element is routing at the level of whole windows inside the high-resolution stream, in place on the full 2D grid. This keeps every pixel present and spatially aligned, which earlier sparse approaches could not do for dense per-pixel prediction.

Position Relative to Prior Work

Two papers motivate this work by locating the problem. U-HRNet shows that high-resolution convolutions consume most of HRNet's budget. It responds by moving blocks to low-resolution branches in a U-shaped design, giving up continuous high-resolution processing. We keep the high-resolution stream intact and instead make its compute uneven across the image.

SHaRPose shows that dense high-resolution processing is wasteful for keypoints. It uses a coarse pass, a decision step, and a fine pass that re-tokenizes keypoint patches, cutting GFLOPs by 25% relative to ViTPose. However, that design is sequential, specific to sparse keypoints, and built on exiting early for easy samples. Our routing happens inside every stage on a fixed grid, so it applies to segmentation. It also adds depth to hard regions rather than removing work from easy ones.

The method itself is borrowed from DND: a nested review pass with shared weights and a blended update. DND reports two difficulties: its scoring module saturated and needed careful control losses, and it was only tested on intermediate layers during fine-tuning. We avoid the saturation problem by selecting a fixed fraction of windows (top-k) instead of thresholding a sigmoid. Vision is the untested domain DND explicitly leaves open.

The backbones come from HRFormer and Lite-HRNet. HRFormer already computes attention within local windows, which makes windows the natural routing unit.

One line of work offers a competing explanation for boundary errors. FDAM argues that self-attention acts as a low-pass filter, so stacking layers blurs boundaries. Under that view, boundary errors would be a filtering artifact rather than a shortage of depth, and a second pass through the same attention could blur further. We test this directly instead of assuming it away.

Research Program

The mechanism works on the high-resolution stream at the end of each stage, after the multi-resolution fusion step. At that point the high-resolution features already contain semantic context from the coarser branches.

The feature map is divided into non-overlapping windows. These are the same windows HRFormer uses for attention; in Lite-HRNet we impose an 8×8 grid. A small scoring module reads pooled features from each window and assigns it a score $s_w$. The top fraction $\rho$ of windows by score are gathered, together with a one-window border of surrounding pixels, so the 3×3 depthwise convolutions see correct neighbors. The gathered windows are run once more through the stage's last block $B$, using its existing weights. The results are written back in place:

$$h_w \leftarrow h_w + s_w\,\big(B(h_w) - h_w\big),$$

Here $h_w$ is the window's feature tensor and $s_w \in (0,1)$ is its routing score. Multiplying by $s_w$ lets gradients reach the scoring module through the selected windows. Unselected windows keep their features unchanged. The only added parameters belong to the scoring module, roughly thousands of weights per stage.

We will study four design questions:

  • Fraction routed. We sweep $\rho$ over 10%, 25%, and 50%.
  • Pass ordering. We compare a review pass placed after fusion against one placed before fusion, which tests whether the second look benefits from fresh low-resolution context.
  • Router supervision. We compare a scoring module trained only by the task loss against one given an auxiliary target: the per-window error of a pass that has not been reviewed.
  • Number of passes. We test whether a second review pass adds anything.

We will also measure, on held-out images, what fraction of selected windows fall on boundaries, thin structures, or keypoint neighborhoods.

Baselines

Unmodified backbone. The naive reference point is the backbone without any review pass, trained with the same schedule. We use HRFormer-T/S for segmentation and Lite-HRNet-18/30 for pose. For context, Lite-HRNet-30 reports 70.4 AP on COCO keypoints. On Cityscapes we found no reported result on Lacuna for Lite-HRNet-30 (the paper only says it beats BiSeNet and ICNet), nor any absolute mIoU on Lacuna for small HRFormer variants. We will therefore retrain these reference points ourselves.

Dite-HRNet-30. The strongest existing lightweight high-resolution method in the file is Dite-HRNet, which adds input-dependent kernels and context modeling to Lite-HRNet. It reports 71.5 AP on COCO keypoints and 87.6 [email protected] on MPII. It is the right competitor because it also adapts computation to the input, but through kernel choice rather than where depth is spent. We will retrain it under our schedule. Beating it means exceeding its AP while using no more parameters and FLOPs within 10%. The review pass can also be stacked on top of Dite-HRNet to test whether the two gains add up.

Larger-scale references. For segmentation, the ceilings are HRNetV2-W48 at 81.1% Cityscapes mIoU and U-HRNet-W48 at 81.9%. These are not fair head-to-head comparisons at our size. Their role is to show how much of the small-to-large gap closes. We will also implement a U-HRNet-style reallocation at small scale with matched GFLOPs, since it is the direct structural alternative.

Ablations of the key ingredient. Each ablation is matched to the review model in FLOPs (within 5%) and trained on identical data, schedule, and tuning budget, with three seeds each:

  • Uniform review: a second pass on all windows of every stage, so routing is absent but weights are still shared. FLOPs are matched by reviewing fewer stages.
  • Random routing: the same fraction $\rho$ of windows, chosen at random.
  • Unshared extra block: the same compute spent on a new block with its own weights.

The proposal succeeds only if the learned-routing review beats all three at matched FLOPs, by more than the seed variance, on boundary-band mIoU.

Why not just…?

Why not just make the network wider or add a few more blocks? Uniform growth is the simplest route to better accuracy. Dite-HRNet's comparison against a uniformly slimmed HRNet (10 AP better at 40% of the GFLOPs) suggests that how capacity is arranged matters more than how much there is, but that does not prove the same holds for added depth. The "unshared extra block" and width-matched baselines settle the question. If they match the review pass at equal FLOPs, the claim fails. The parameter argument would then reduce to a few hundred thousand weights, which matters little at this scale, and we would report the result as negative.

Why not just select windows with an edge detector instead of a learned router? This is a real threat. If boundaries are where errors live, a Sobel filter on the image, or ground-truth boundaries as an upper-bound oracle, might choose windows as well as a trained scorer and with no training instability. We will run both, with $\rho$ fixed. If Sobel routing matches learned routing, the contribution simplifies to "review edge windows," which is still useful and simpler. Learned routing earns its place only if it also selects windows that are hard for reasons other than edges, such as occluded keypoints or ambiguous classes. Our selection-composition analysis would show this.

Why not just fix boundary blurring with a high-pass correction such as FDAM? FDAM reports +2.4 mIoU for SegFormer-B0 on ADE20K by inverting attention's low-pass behavior. If boundary errors come from spectral smoothing, a second pass through a smoothing operator should help little or even hurt. We will measure the high-frequency energy of reviewed windows before and after the pass. We will also train three variants of HRFormer: FDAM alone, review alone, and both. If FDAM alone captures the gain, our depth-shortage hypothesis is wrong. If the two gains add up, they address different failure modes.

Why not just use SHaRPose-style coarse-to-fine token splitting? It is proven for pose, at 77.4 AP for SHaRPose-Base. However, it requires a sequential coarse pass and a decision step, and it targets sparse outputs. We compare against it on COCO keypoints at matched GFLOPs and frames per second. On segmentation it has no direct counterpart, which is where our design is meant to apply.

Proposed Experiments

Segmentation. We train HRFormer-T/S with and without review on Cityscapes, ADE20K, and PASCAL-Context. Alongside standard mIoU, we report boundary-band mIoU, computed within a few pixels of label boundaries derived from the annotations. The prediction is that gains appear mostly in the boundary band and are small in region interiors. If gains are instead spread uniformly, the "hard windows" story is wrong even if the averages improve.

Thin structures. ThinObject-5K and HRSOD, scored with $IoU_{thin}$, are the stress test for whether a second look recovers 1-pixel structures that TOS-Net needed a dedicated full-resolution edge stream to capture. We use the segmentation setting and do not claim parity with interactive TOS-Net.

Pose. COCO keypoints and MPII use Lite-HRNet-18/30 with the review pass, compared against Dite-HRNet-30 at 71.5 AP.

Cost reporting. All results report parameters, FLOPs, and measured GPU latency. Gathering and scattering windows is irregular memory access, the hardware concern the file raises. A FLOP win that disappears in wall-clock time will be reported as such.

Decisive check. The key result is the matched-FLOP comparison of learned routing against uniform review, random routing, unshared blocks, and Sobel routing on Cityscapes boundary-band mIoU. The idea stands only if learned routing beats the first three and at least matches the fourth.