Networks that label every pixel of an image are expensive. This includes segmentation networks, which assign a class to every pixel, and pose estimators, which locate body joints. The best of them keep a full-detail, high-resolution feature map through the whole network instead of shrinking the image and upsampling it back later. The cost is that every location in that map gets the same number of feature channels. A patch of flat sky receives as much computation as a bicycle spoke. One prior method, adaptive exiting, lets a pixel stop computing once the network is confident about it. That decision is binary: a pixel either gets full width or nothing. Separately, Matryoshka representation learning trains vectors so that their first few dimensions already carry coarse meaning and later dimensions add fine detail. It has only ever been applied to whole-image vectors.
This note proposes combining the two. Train the high-resolution stream of a dense-prediction backbone so that its channels are ordered from coarse to fine. Then, in each block, update only the first $d_i$ channels at location $i$, with $d_i$ chosen per pixel from a small set of widths according to how hard that pixel currently looks. The claim to test is empirical and falsifiable. Pixel difficulty is graded rather than all-or-nothing. If so, a graded per-pixel channel budget should beat a binary skip mask at the same compute, especially on thin structures and object boundaries. It should also beat simply using a uniformly narrower high-resolution network.
Core Claim and Mechanism
High-resolution backbones spend most of their cost on per-location channel mixing. Lite-HRNet found that the 1×1 pointwise convolutions across its parallel streams dominate compute. Lite-HRNet replaces them with cheaper conditional channel weighting, but channel widths stay fixed at inference.
These per-location operations are also the ones that can be made heterogeneous without breaking dense hardware execution. Tokens assigned to the same width can be gathered into a dense matrix, multiplied by the top-left $d \times d$ block of the weight, and scattered back. This avoids the irregular sparse convolutions that Anytime Dense Prediction with Confidence Adaptivity (ADP-C) needs for its per-pixel skipping.
The key design choice is that the representation at every location stays full width; only the update is truncated. For block $l$ with transformation $f_l$ and hidden state $h_l(i) \in \mathbb{R}^D$ at location $i$, the update is
$$ h_{l+1}(i)_{[1:d_i]} = h_l(i)_{[1:d_i]} + f_l\big(h_l\big)(i)_{[1:d_i]}, \qquad h_{l+1}(i)_{[d_i+1:D]} = h_l(i)_{[d_i+1:D]}, $$
where $d_i \in \{D/16, D/8, D/4, D/2, D\}$ is the width tier assigned to location $i$ at that block. Channels above $d_i$ simply pass through the residual connection unchanged. Because every token still carries a complete vector, the operations that mix neighbouring tokens always see well-defined inputs. These are windowed attention and the depthwise convolutions in HRFormer's feed-forward layers, and they remain full width because they are cheap. This sidesteps the interpolation ADP-C must do for masked-out pixels.
The width tier comes from the prediction entropy of a lightweight auxiliary head that reads only the first $D/16$ channels. It is calibrated to a target average budget. A small learned router is tested as a variant.
Training follows Matryoshka Representation Learning (MRL), which supplies the method: a nested loss over prefix widths with shared weights. The dense version differs in one respect. Widths are sampled per token rather than per image, so the network learns to function when a narrow-width sky pixel sits next to a full-width edge pixel. The training objective is
$$ \mathcal{L} = \sum_{d \in \mathcal{D}} \lambda_d \, \ell\big(y, \hat{y}_d\big) + \ell\big(y, \hat{y}_{\pi}\big), $$
where $\hat{y}_d$ is the prediction with all tokens at width $d$, $\hat{y}_{\pi}$ is the prediction under a random per-token width assignment $\pi$, $\ell$ is the task loss, and $\lambda_d$ are MRL-style weights.
Relation to Prior Work
Four lines of work bound this proposal.
First, high-resolution backbones motivate it. HRNet keeps parallel multi-resolution streams, HRFormer runs local-window attention within them, and UViT keeps a single 1/8-scale map with a constant large embedding. All three fix channel width per stream and location. That uniform allocation is what we relax.
Second, adaptive computation constrains the design. ADP-C shows large interior regions become confident early; it reports a 44.4% FLOP reduction on Cityscapes at unchanged accuracy. Its decision is binary, and its sparse execution yields only a 1.48× CPU speedup. ThinkingViT nests attention heads and widens the whole network when global entropy is high, so its elasticity is per image, not per location.
Third, MRL supplies the nesting technique but was evaluated on pooled global vectors. Matryoshka Multimodal Models nests along the number of spatial tokens while keeping channels fixed. We nest along channels while keeping every spatial location.
Fourth, PointRend and TOS-Net show that boundaries and 1-pixel-wide structures need special treatment. PointRend works post hoc on top of a backbone; TOS-Net adds a dedicated full-resolution edge stream. Our mechanism places the extra capacity inside the backbone, only where it is needed.
Research Program
The first stage is diagnostic, and it decides whether the rest is worth doing. We train a nested HRFormer-B on Cityscapes with the dense Matryoshka objective. For each pixel we then record the smallest prefix width at which its prediction matches the full-width prediction. The shape of this distribution determines the answer.
If it is bimodal, the graded idea collapses into ADP-C. Bimodal here means almost all pixels are resolved at $D/16$ and the rest need all of $D$. We would report that as a negative result. If a substantial mass sits at intermediate widths, and that mass concentrates on boundary bands and thin structures, graded allocation has room to beat binary skipping. We expect the latter, since MRL's finding that fine distinctions live in later dimensions should hold spatially. The data will settle it.
The second stage builds the tiered execution with gather–GEMM–scatter kernels and measures real GPU latency, not only FLOPs. The third stage compares entropy-based and learned width assignment, and asks whether the width map should be fixed after the first few blocks or reassigned at each stage.
Baselines
The naive reference is a uniformly narrower high-resolution backbone at the same average compute. This is HRFormer with its high-resolution stream width reduced until its FLOPs equal our target budget, trained with the identical recipe, data and schedule. In the lightweight regime, Lite-HRNet is the strongest instance of this idea; it reports 70.4 AP on COCO pose at 0.7 GFLOPs. For the full-capacity upper reference, HRFormer-B reports 82.6% mIoU on Cityscapes and HRNetV2-W48 reports 81.1%.
The strongest existing method on the actual question is ADP-C. It reports matching base HRNet accuracy on Cityscapes with 44.4% fewer FLOPs, and on MPII with 59.1% fewer. We will reimplement its confidence masking on our exact backbone, so that the only difference is binary skipping versus graded widths. We will then sweep its threshold to trace a full accuracy–FLOPs curve alongside ours. Beating it means a curve that dominates ADP-C's at matched FLOPs, with the gap concentrated in boundary-band mIoU, and equal or lower measured GPU latency.
The per-image elastic baseline follows ThinkingViT. The same nested backbone chooses a single width for the whole image from global entropy. This isolates whether spatial heterogeneity, rather than nesting itself, produces any gain. The file gives no dense-prediction numbers for ThinkingViT (no reported result found on Lacuna for ThinkingViT on Cityscapes or ADE20K), so we will run it ourselves.
Three ablations isolate the key ingredient. The first runs graded widths on a backbone trained without the nested loss, truncating ordinary channels. The second assigns tiers at random at the same budget. The third uses an oracle assignment from the diagnostic stage to bound the achievable gain.
All methods share data, augmentation, schedule, and the same tuning budget for thresholds or widths.
Why not just…?
Why not just skip confident pixels, as ADP-C does? This is the real threat. If pixel difficulty is essentially binary, ADP-C already captures most of the savings, with a simpler decision. The diagnostic distribution in stage one and the head-to-head accuracy–FLOPs curves are designed to settle this. We commit to reporting it if graded widths add nothing beyond binary skipping. One independent advantage remains even in that case. Our pass-through updates keep every token's vector defined and execute as dense matrix multiplies. ADP-C, by contrast, needs interpolation and sparse kernels, and gets only 1.48× CPU speedup.
Why not just train a narrower high-resolution network? Lite-HRNet shows that thin, uniformly narrow high-resolution streams can be strong. A uniform narrow width, however, starves exactly the pixels that TOS-Net shows are hardest: 1-pixel-wide structures, where it reports gains of roughly 30% in $IoU_{thin}$ from dedicated capacity. The matched-FLOPs narrow baseline tests whether redistributing channels beats shrinking them everywhere. The test is sharpest on thin-structure and boundary metrics.
Why not just choose one width per image, like ThinkingViT? Street scenes almost always contain both large flat regions and fine structures. A per-image choice therefore must pick full width for nearly every image, and saves little. The per-image baseline measures this directly.
Why not just add PointRend to a cheap backbone? PointRend states that it cannot fix structures the backbone has already lost. Thin objects erased early in the network give its point head nothing to query. We include a narrow backbone with PointRend at matched compute. Our method should win on $IoU_{thin}$ and could plausibly tie on ordinary boundary quality; either outcome is informative.
Proposed Experiments
The primary benchmark is Cityscapes semantic segmentation. We report mIoU and mIoU restricted to a band a few pixels wide around ground-truth boundaries, plotted against FLOPs and measured GPU latency. MPII and COCO pose test whether the gains transfer to keypoints, compared against ADP-C's 59.1% MPII reduction and Lite-HRNet's 70.4 AP. ThinObject-5K and HRSOD, scored with $IoU_{thin}$, test the thin-structure hypothesis directly. We train non-interactively, so TOS-Net's interactive 65.1% is context rather than a matched baseline.
Three checks are decisive. The width-requirement distribution must be non-bimodal. The graded curve must dominate ADP-C's on boundary-band mIoU at matched FLOPs. Random tier assignment and the non-nested ablation must both lose most of the gain, which would show that the benefit comes from difficulty-aware allocation over coarse-to-fine ordered channels, not from width reduction alone. Finally, visualizing the width maps should show wide tiers tracing edges and thin objects. If they do not, the mechanism is not doing what we claim.
References
- Lite-HRNet: A Lightweight High-Resolution Network
- Anytime Dense Prediction with Confidence Adaptivity
- Matryoshka Representation Learning
- Deep High-Resolution Representation Learning for Visual Recognition
- HRFormer: High-Resolution Transformer for Dense Prediction
- A Simple Single-Scale Vision Transformer for Object Detection and Instance Segmentation
- ThinkingViT: Matryoshka Thinking Vision Transformer for Elastic Inference
- Matryoshka Multimodal Models
- PointRend: Image Segmentation As Rendering
- Deep Interactive Thin Object Selection