Networks for dense prediction tasks such as pose estimation and segmentation often keep a full-resolution feature map alive through the whole network instead of shrinking the image early. HRNet-style models do this with a high-resolution branch that runs in parallel with coarser branches. The cost is large intermediate tensors that must be written to and read from memory at every layer, and on phones and accelerators that memory traffic is often the real bottleneck. A separate line of compression work removes "lazy" nonlinearities, meaning activation functions that pass almost everything through unchanged, and then merges the now-adjacent linear layers into one. This is linear layer fusion: if nothing nonlinear sits between $W_2$ and $W_1$, one precomputed $W_2W_1$ does the same job.
This proposal combines the two by noting where the savings from fusion actually land. Fusing two layers removes one intermediate activation tensor, and that tensor is largest in the high-resolution branch. The weights of the fused kernel do not grow with image size. Existing evidence, however, says nonlinear redundancy is concentrated in late, low-resolution, semantic layers. That is exactly where fusion saves the least memory traffic. We propose to measure, stream by stream, how much nonlinearity a high-resolution branch really needs. We would then build a linearization method whose budget is allocated by resolution-dependent memory cost, and test whether fine spatial detail survives, using boundary and thin-structure metrics rather than mean accuracy alone.
Core Claim
In parallel multi-resolution backbones, the high-resolution stream mostly does local, near-linear filtering, so a large fraction of its activations can be removed and the adjacent convolutions folded. Because the folded-away tensors scale with spatial area, this yields much larger latency and memory savings per unit of accuracy lost than linearizing the low-resolution streams. Two outcomes would make the claim new. The first is a measured per-stream redundancy profile, which no prior work reports. The second is an allocation rule under which a resolution-aware method beats a stage-agnostic one on the same hardware budget. If the high-resolution stream turns out to need its nonlinearities for thin structures, the claim is wrong, and we would rather learn that clearly.
Relation to Prior Work
The fusion methods supply the mechanism. DepthShrinker searches for removable activations in inverted-residual blocks and merges expansion, depthwise and projection convolutions into one dense convolution. It reports up to 88.2% block latency reduction on an RTX 2080Ti, but only on classification backbones that treat all stages alike. HetDPT shows that FFN GELUs in vision transformers can be removed and the two FFN matrices merged with almost no loss: 1.58× speedup on DeiT-B at 81.8% top-1. It works only on models that have already downsampled. Neither looks at a backbone in which a full-resolution stream coexists with coarse ones. We reuse their merging algebra and replace their uniform criterion.
The redundancy-measurement papers motivate the question and also complicate it. EASIER and NEPENTHE use the entropy of each neuron's on/off state to find layers that behave linearly, and both find such layers mostly near the output. Nonlinear Advantage shows a ResNet50 can be made 75% linear on ImageNet with almost no accuracy loss, and argues that effective depth tracks task difficulty. Their placement finding cuts against us: if redundancy lives only in semantic layers, the high-resolution stream may be the worst place to linearize. That is an empirical question, and our first experiment answers it.
The high-resolution architectures define the regime and the evaluation. Lite-HRNet and HRFormer keep parallel streams but leave every activation in place. TOS-Net shows that one-pixel structures are destroyed by downsampling, which is why boundary metrics are the right test of whether linearization damages detail. EfficientViT is a cautionary contrast. It "linearizes" attention in a different, kernel-based sense, and then has to add depthwise convolutions back to recover local detail. That suggests local high-frequency processing is fragile, which our experiments must take seriously.
SESR's collapsible linear blocks are the closest three-way connection. They train with extra linear convolutions and collapse them for high-resolution super-resolution. That is training-time overparameterization of a single-stream network built from scratch. It is not post-training removal of existing nonlinearities, and it does not address allocation across parallel streams.
Research Program
The first phase is measurement. For pretrained Lite-HRNet-30, HRNet-W32/W48 and HRFormer-B, we would record per-activation on/off entropy (following EASIER) on calibration images, reported separately for each resolution stream and depth. We would also compute per-stream path-length statistics in the spirit of Nonlinear Advantage. The output is a map of nonlinear redundancy indexed by resolution, which the literature does not contain.
The second phase is a cost model that prices each removable activation by what folding actually saves on the target device. Removing activation $l$ in stream $r$ and folding its neighbours eliminates an intermediate tensor of roughly $H_r W_r C_l b$ bytes. Here $H_r W_r$ is that stream's spatial size, $C_l$ the channel count at that point, and $b$ bytes per element. The fused kernel, however, can raise multiply-accumulate count. Folding two $3\times3$ convolutions gives a $5\times5$ kernel with about $25/18$ the operations, and folding into a depthwise convolution makes it dense. The net benefit is
$$\Delta T_l \approx \frac{H_r W_r C_l b}{\beta} - \frac{\Delta M_l}{\pi},$$
where $\beta$ is effective memory bandwidth, $\Delta M_l$ the added multiply-accumulates, and $\pi$ effective compute throughput. We calibrate $\beta$ and $\pi$ by profiling single blocks on each device rather than trusting the formula. Because the first term scales with image area and the second does not, folding is expected to pay off in high-resolution streams and possibly to cost time in low-resolution ones.
The third phase is allocation. We would run a DepthShrinker-style differentiable activation mask with a penalty weighted by measured $\Delta T_l$, then fold. Residual identities are absorbed into the fused kernel's center tap, and the layer-size threshold from Linearity-based neural network compression is imposed so that folding never inflates parameters. A short fine-tune follows, matching HetDPT's observation that activation removal recovers quickly while attention removal does not.
Baselines
The do-nothing reference is the unmodified pretrained backbone. Lite-HRNet-30 reports 70.4 AP on COCO pose at 0.7 GFLOPs (Lite-HRNet), and HRFormer-B reports 82.6% mIoU on Cityscapes (HRFormer). We would re-measure both with our own pipeline and devices, so that every other method is compared against the same numbers.
The strongest competitor is DepthShrinker's search applied unchanged to these backbones, with its own latency-aware objective measured on the same device. This is the real threat. A good latency-aware search might discover the resolution asymmetry by itself, with no explicit cost model. No reported result was found on Lacuna for DepthShrinker on HRNet-family models, so we run it ourselves with the same calibration data, the same fine-tuning epochs and the same number of search iterations as our method. For HRFormer we also run HetDPT's FFN-merging pipeline from its released code (Efficient-AI-for-All/HetDPT) under the same budget. The anchor from that paper is its reported 1.58× speedup at 81.8% on DeiT-B, a classification setting rather than ours.
The ablations isolate the key ingredient. Entropy-only allocation, as in EASIER, has no cost weighting. Reversed allocation forces removals into the low-resolution streams at a matched latency target. Uniform allocation removes the same fraction from every stream. Beating the baselines means a better accuracy-versus-measured-latency curve and a better boundary-metric-versus-latency curve, both across at least two devices. If uniform latency-aware DepthShrinker lands on the same curve, the method contribution collapses and the paper becomes a measurement study.
Why not just…?
Why not just shrink the high-resolution stream's width, or feed it a smaller image? Width pruning is the standard answer to oversized tensors. On ImageNet, DepthShrinker reports 3.06% higher accuracy at 1.53× throughput over MetaPruning (DepthShrinker), but that evidence comes from classification, not dense prediction. Lowering input resolution is cheaper still, yet it is exactly what TOS-Net shows destroys thin structures (TOS-Net). Both are run as baselines at matched latency, and the boundary metrics will show whether linearization keeps detail that width cuts or downscaling lose.
Why not just let a latency-aware search find the allocation? This may work as well as our method, and we say so. The resolution-aware cost model is cheap to add, but its value is only proven if it beats measured-latency DepthShrinker at equal search budget. Even if it does not, the per-stream redundancy map and the evidence that high-resolution streams tolerate or resist linearization remain the main scientific result.
Why not just fine-tune the pruned model longer, or with low-rank adapters? HetDPT reports that activation removal recovers almost fully after brief fine-tuning, so long healing should not be needed. Low-rank adapters add branches unless merged. We report accuracy as a function of fine-tuning epochs. As a fallback for high removal rates, we consider RestoreLCC's approach of absorbing a correction derived from the activation gap into bias terms; it has been shown only in LLMs.
Proposed Experiments
We evaluate COCO pose (AP) with Lite-HRNet-30 and HRNet, and Cityscapes segmentation (mIoU plus mIoU within thin bands around ground-truth boundaries) with HRNet-W48 and HRFormer-B. For fine detail, a segmentation head is trained on ThinObject-5K and scored with $IoU_{thin}$. TOS-Net's 65.1% $IoU_{thin}$ on HRSOD is an interactive-selection result and only a rough reference, not a comparable baseline. Each model is profiled for latency, peak memory and activation bytes on a desktop GPU and a mobile GPU or NPU.
The first decisive check is the measurement itself. If high-resolution activations show entropy as high as low-resolution ones and removing them sharply lowers boundary-band mIoU and $IoU_{thin}$, the core hypothesis fails. The second is the cost-model check: predicted $\Delta T_l$ must rank-correlate with measured block latency, or the allocation rule has no basis. The third is the head-to-head. At matched measured latency, resolution-aware allocation must beat uniform, reversed, entropy-only and latency-aware DepthShrinker allocation on both mean and boundary metrics, on both devices. The prediction is that the gap widens as input resolution grows, because the memory term scales with image area.
References
- DepthShrinker: A New Compression Paradigm Towards Boosting Real-Hardware Efficiency of Compact Neural Networks
- Rethinking Depth Pruning for Vision Transformers: A Heterogeneity-Aware Perspective
- The Simpler The Better: An Entropy-Based Importance Metric To Reduce Neural Networks' Depth
- NEPENTHE: Entropy-Based Pruning as a Neural Network Depth's Reducer
- Nonlinear Advantage: Trained Networks Might Not Be As Complex as You Think
- Lite-HRNet: A Lightweight High-Resolution Network
- HRFormer: High-Resolution Transformer for Dense Prediction
- Deep Interactive Thin Object Selection
- EfficientViT: Multi-Scale Linear Attention for High-Resolution Dense Prediction
- Collapsible Linear Blocks for Super-Efficient Super Resolution
- Linearity-based neural network compression
- Restoring Pruned Large Language Models via Lost Component Compensation