Backbone-Agnostic Dilated Side Networks for Dense Prediction

Large pretrained vision models come in incompatible shapes: convolutional networks, Vision Transformers, and newer state space models such as Mamba. Most methods for cheaply adapting them to pixel-level tasks like segmentation and detection are built for one shape only. Plain Vision Transformers, for example, get adapters that rewire their attention blocks. We propose a lightweight "side network" that never touches the backbone's weights or internals. It only reads a few intermediate feature maps through forward hooks and passes no gradients back through the backbone. In parallel, it looks at the raw high-resolution image through a compact stack of multi-scale dilated convolutions. A dilated convolution is a convolution whose kernel taps are spread apart, so it sees a wide area without extra parameters or downsampling. The side network fuses what it sees with the frozen backbone's features and outputs a feature pyramid for any standard dense prediction head.

The bet is that two properties can be had at once. Because the side network reads backbone features only through hooks, it needs no backpropagation through the backbone: one recipe runs unchanged on CNNs, ViTs, and SSMs, and many tasks can share one frozen backbone instance. Because its own path sees the image at full resolution through a cheap multi-scale dilation lattice, it can supply the fine spatial detail and receptive field that frozen features lack. That is exactly what prior evidence says frozen-backbone adaptation is missing for dense tasks, especially when labeled data are scarce.

Core Claim

A side network of about one to two million parameters can match or beat backbone-specific dense adapters and full fine-tuning in the low-data regime, at a fraction of the training memory, on three backbone families with identical hyperparameters. Its design pairs hook-only reads of a frozen backbone with a high-resolution image path built from channel-split dilated convolutions. The new ingredient is the image path, not the decoder. The program is built to falsify this: if a decoder of equal size on the backbone's tapped features alone does as well, the claim fails.

Where This Sits

Four papers motivate the gap. Side-Tuning blends a frozen backbone's output with a side network's output via $R(x)=\alpha B(x)+(1-\alpha)S(x)$, where $B$ is the frozen base, $S$ the side network, and $\alpha$ a blending weight. UniPT guides a parallel network with the backbone's final output and cuts T5-base training memory on GLUE from 17.6GB to 2.9GB. Both are architecture-agnostic, but both work on vector representations and were evaluated on classification, language understanding, and retrieval, never on segmentation or detection. Dense adapters take the opposite trade. ViT-Adapter injects a convolutional spatial prior into ViT blocks through cross-attention and is fine-tuned end to end. Hierarchical Side-Tuning inserts a learnable token into every ViT block and unfreezes the backbone's normalization layers, which breaks weight sharing across tasks.

Two related methods share this coupling. Side Adapter Network runs a high-resolution ViT side path but writes attention biases into CLIP's self-attention. BarLeRIa states it cannot run on CNNs. We keep Side-Tuning's black-box contract and import Side Adapter Network's idea of a separate high-resolution image path. Unlike any of these, we make that path convolutional and dilated, and we never write into the backbone.

The multi-scale literature supplies the method. CFPNet splits channels into parallel paths with different dilation rates and factorized $3\times1$/$1\times3$ kernels, then fuses them hierarchically to suppress gridding artifacts, the checkerboard holes a single dilation rate leaves. It reaches 70.1% class mIoU on Cityscapes with 0.55M parameters. PSConv cycles dilation rates across channels inside one layer at zero parameter cost, and adds 2.1 box AP to a ResNet-50 Mask R-CNN on COCO. We reuse these blocks as the side network's image path; neither paper uses them for adapting a frozen model.

Two papers constrain the design and appear to pull in opposite directions. Dual-Kernel Adapter finds that standard adapters shrink the effective receptive field under severe data scarcity. It fixes this with a $51\times51$ depthwise branch inside ViT blocks, which beats full fine-tuning on BRATS at 1.25% data but forced batch size 1 for segmentation. That tells us receptive field matters and that big dense kernels are too memory-hungry. Is PEFT Enough for Cell Segmentation? finds that on LIVECell every adapter and full fine-tuning sit within a hair of 0.56 AP. Its authors conclude that instance separation "cannot simply be patched in later." We take this negative result seriously. Their adapters all modify backbone features. Ours adds an independent high-resolution pixel path, which is the one thing that could plausibly supply information the backbone discarded. LIVECell is therefore our sharpest test, and a null result there would be informative rather than embarrassing.

Research Program

The backbone interface is deliberately minimal. For any backbone we register forward hooks at four depths, roughly quarter points of the network. Token outputs are reshaped to a 2D grid when the backbone is a ViT or SSM; CNN stage maps are used as is. Each tapped tensor is detached, so no gradient enters the backbone. Each tap is resampled to strides 8, 16, and 32 and projected by a per-tap $1\times1$ convolution to a shared width. The per-tap projection is the only backbone-specific component, and its size depends only on the tap's channel count.

The image path runs on the input at stride 4 and 8. It uses CFPNet-style channel-pyramid blocks: channels are split into groups with dilation rates such as 1, 2, 4, and 8, kernels are factorized, and the groups are fused hierarchically. The blocks are interleaved with PSConv-style cyclic-dilation layers. Fusion at each stride is a gated addition of the projected backbone tap and the image-path feature, with the gate predicted per channel from both inputs. This replaces Side-Tuning's single $\alpha$ and is cheaper than cross-attention. The output pyramid feeds unmodified heads: UPerNet for segmentation, Mask R-CNN or Cascade Mask R-CNN for detection, and the original instance head for LIVECell.

We will build this on three backbone families: a ResNet or ConvNeXt-style CNN, a plain ViT, and a vision Mamba model. All three use identical side-network hyperparameters, chosen once on a Cityscapes validation split with the ViT and then frozen. We will also run a backbone-swap test. A side network trained on one backbone version is moved to another, for example a ViT replaced by a larger or updated checkpoint in the spirit of Adapt Once, Thrive with Updates and A4A. Only the per-tap projections are retrained, and we measure how much of full side-network retraining this recovers.

Baselines

The naive reference point is a frozen backbone with only the dense head trained on its last-layer features, a dense analogue of linear probing. On COVID chest X-ray at 0.63% data, linear probing already beat standard adapters (86.84% vs 83.29% accuracy) in Dual-Kernel Adapter, so it is not a strawman.

The most dangerous baseline is the taps-only decoder. It receives the same four hooked backbone features and the same per-tap projections, followed by a feature pyramid decoder whose parameter count and FLOPs are matched to our full side network, but it has no image path. It shares our architecture-agnosticism and memory profile, so any win we report must come from the image path. Beating it means a consistent gain across all three backbone families beyond seed variance, with a larger gain at 1–10% data than at full data.

The receptive-field ablation swaps our dilated image path for one of equal parameter count built from plain $3\times3$ convolutions. A second variant uses DKA-style $51\times51$ plus $5\times5$ depthwise branches. Together these isolate whether dilation specifically matters and what it buys in memory.

Among backbone-specific methods, the strongest external competitors are Hierarchical Side-Tuning on ViT and Dual-Kernel Adapter in low-data medical segmentation. HST reports 49.7 box AP and 43.2 mask AP on COCO test-dev with Cascade Mask R-CNN, 0.8 box AP above full fine-tuning. DKA reports 23.02% mIoU on BRATS at 1.25% data against 22.39% for full fine-tuning and 18.95% for standard adapters. ViT-Adapter-L, at 60.9 box AP and 61.5 ADE20K mIoU, is reported as a fully fine-tuned upper reference, not a like-for-like competitor. We found no reported HST result on ADE20K on Lacuna. Full fine-tuning and LoRA with the same head round out the comparison.

On LIVECell the reference numbers come from the no-go paper: 0.5447 AP for base Cellpose-SAM, 0.5600 for the best adapter, and 0.5627 for full fine-tuning. All methods share data splits, heads, schedules, and a fixed hyperparameter search budget per method. We report peak training memory and throughput alongside accuracy. Where we rerun HST and DKA from their released settings (DKA's code is at misswayguy/DKA), we use their recommended learning rates rather than ours.

Why not just…?

Why not just put a feature pyramid decoder on the frozen backbone's intermediate features? This is the real threat, and it is why the taps-only decoder is our main baseline rather than an afterthought. Frozen features from a stride-16 ViT, or from a backbone that never saw dense supervision, may simply lack the boundary detail that the LIVECell no-go result points to. No decoder can recover what was never encoded. If the matched taps-only decoder ties our method across backbones and data fractions, we will report that the image path is unnecessary. The contribution then shrinks to a cross-architecture decoder study, and we will not argue around it.

Why not just use LoRA or bottleneck adapters, or fine-tune everything? These methods must backpropagate through the backbone, so training memory stays near full fine-tuning; UniPT documents the gap side tuning closes. They also have to be re-engineered for each backbone family, as Parameter-Efficient Fine-Tuning of State Space Models shows for Mamba. In low-data regimes they underperform: LoRA scored 80.43% against linear probing's 86.84% in DKA's COVID experiment. Full fine-tuning also forfeits sharing one backbone across tasks. Our memory and accuracy comparison against both settles this directly.

Why not just use DKA's large kernels, or HST, which already work? Both are tied to ViT internals, and HST unfreezes normalization layers inside the backbone. DKA's $51\times51$ kernels forced batch size 1 for segmentation. The large-kernel variant of our side network tests whether dilation reaches the same receptive field at lower memory. Running both baselines on the ViT, where they are native, ensures we are not comparing on our own turf.

Why not just train CFPNet from scratch on the target task? At full Cityscapes data this may be competitive, which is why CFPNet alone is included as a backbone-free reference. The claim is that the frozen backbone matters most at 1–10% labels, so the gap to scratch-trained CFPNet should widen as data shrink. If it does not, the backbone is not earning its cost.

Proposed Experiments

Experiments run on Cityscapes and ADE20K semantic segmentation, COCO detection and instance segmentation, BRATS at 1.25% data, and LIVECell. Every setting uses all three backbone families and data fractions of 1%, 10%, and 100% where the dataset allows. We report mIoU or AP, peak training memory, trainable parameters, and inference throughput, each over three seeds.

The first decisive check is the gap between our side network and the taps-only decoder at 1% and 10% data, averaged over backbones; the claim needs it positive on every family. The second is BRATS at 1.25%, where we need to reach or exceed DKA's 23.02% mIoU at a batch size above 1 and lower memory. The third is LIVECell, where any result clearly above the 0.5627 AP full fine-tuning ceiling would be the first evidence that a separate pixel path can add what that paper argued cannot be patched in. The fourth is the backbone-swap test: what fraction of full retraining performance projection-only retraining recovers. Finally, the dilated-versus-plain ablation tells us whether the multi-scale dilation lattice, rather than mere extra convolution, carries the effect.