One Shared Dilated Adapter for Data-Scarce Fine-Tuning

When a large pretrained vision model is adapted to a new medical task with only a few hundred labelled images, the usual lightweight add-on modules can make things worse than adding nothing at all. The Dual-Kernel Adapter study traced this failure to a shrinking effective receptive field, meaning the image region that actually influences each output. It fixed the problem with a very large $51 \times 51$ convolution inside each adapter (Dual-Kernel Adapter). That fix is expensive in memory; the authors had to train segmentation with batch size 1. We propose a cheaper route to the same spatial reach. A single small adapter is shared by every layer of the frozen transformer. It uses $3 \times 3$ dilated convolutions, which are kernels with gaps between their taps so they cover a wide area with only nine weights. The pattern of dilation rates shifts from layer to layer.

The idea fits in two sentences. Tying the adapter's weights across depth cuts the trainable parameters to one layer's worth and regularizes it, which suits tiny datasets. Changing the dilation pattern at each layer tells the shared module which depth it is at without adding any parameters, and stacking those different patterns fills in the holes each dilated kernel leaves on its own.

Core Claim

We claim that under extreme data scarcity (roughly 1% of the training set), a weight-tied adapter with a layer-shifted dilation lattice will match the Dual-Kernel Adapter's accuracy. It should do so with at least an order of magnitude fewer trainable adapter parameters and with peak training memory small enough to allow batch sizes above 1 for segmentation. We also claim the layer-shifted dilation is doing real work. A tied adapter with a fixed dilation pattern should lose accuracy and receptive field, because a looped module that applies the identical function at every depth cannot adjust its behaviour to where it sits. The looped-transformer literature identifies exactly this weakness and fixes it with timestep encodings (On Expressive Power of Looped Transformers). Our dilation shift is a spatial, parameter-free version of that timestep signal.

Concretely, at transformer block $l$ with patch-token map $x_l$, the adapter output is

$$h_l = x_l + s_l \, U\,\sigma\!\big(\mathrm{DW}_{\pi_l}(D\,x_l)\big),$$

where $D$ and $U$ are the shared down- and up-projections, $\sigma$ is a nonlinearity, and $\mathrm{DW}_{\pi_l}$ is a shared depthwise $3 \times 3$ convolution. That convolution's channels are assigned dilation rates by the pattern $\pi_l$, a cyclic shift by $l$ of a base pattern such as $(1,2,1,4,1,8)$. The only per-layer quantity is $s_l$, a learned scalar gate. All kernel weights are identical across layers; only which channel sees which dilation changes.

How This Relates to Prior Work

The proposal draws on four groups of prior work.

The motivating evidence comes from Dual-Kernel Adapter (also indexed as art_2ac35ce0; Dual-Kernel Adapter: Expanding Spatial Horizons for Data-Constrained Medical Image Analysis). It pairs untied, per-layer $51 \times 51$ and $5 \times 5$ depthwise branches with an asynchronous learning-rate schedule. It shows that spatial reach, rather than parameter count, separates adapters that work from those that fail. We keep its diagnosis and its benchmarks. We replace its physical large kernels with dilation, and its per-layer parameters with one tied module.

The multi-scale machinery comes from compact dilated designs. PSConv cycles dilation rates across channels and shifts them per output filter. This gains multi-scale context with zero extra parameters, and a dilated Winograd kernel brings the runtime overhead down to 1.06x. CFPNet assigns dilation rates to channel groups and fuses them hierarchically to suppress gridding, the checkerboard gaps that dilated kernels leave. Multi-Scale Context Aggregation by Dilated Convolutions establishes that dilation expands the receptive field without extra weights. All three work inside forward CNNs. None has been tried as an adapter on a frozen transformer, and none shifts its lattice across depth as a depth signal. Our shift across layers plays the role that PSConv's shift across filters plays within a layer.

The weight-tying machinery comes from recursive transformers. Sliced Recursive Transformer loops shared attention blocks and needs extra projection layers and learnable residuals to avoid collapse. It relies only on self-attention and is tested on classification. ALBERT-based systems report that tying acts as a regularizer on small datasets (YNU-HPCC at SemEval-2021 Task 6). We apply tying to the adapter rather than the backbone, and our gate $s_l$ is a minimal stand-in for SReT's learnable residuals.

Dense-prediction adapters act as constraints on the design. ViT-Adapter and META inject spatial priors with per-block modules. META shows that memory-access cost, not just parameters, dominates adapter overhead, and it keeps its convolution branch at $1 \times 1$. This constraint pushes us toward a design that is cheap in memory traffic, not just in weights, so memory and latency are measured outcomes and not afterthoughts.

Research Program

We build the tied poly-dilated adapter on the same frozen backbone and task heads used by the Dual-Kernel Adapter study. We reuse its asynchronous learning-rate schedule so that the adapter design is the only change. The study then proceeds in three strands.

The first strand measures the effective receptive field directly. We compute input gradients with respect to the centre output token and do so for every variant. This tests whether the shifted lattice restores the spatial reach that standard adapters lose, and whether gridding holes remain visible when the dilation pattern is fixed rather than shifted.

The second strand separates the two ingredients. We cross tied versus untied weights with shifted versus fixed dilation. We add a tied adapter that uses a learned per-layer embedding instead of the dilation shift, which directly implements the timestep-encoding alternative.

The third strand is a scaling check. We vary the data fraction from 0.63% up to 100%. We expect tying to help most at the smallest fractions and possibly to hurt at full data, where capacity rather than regularization is the binding constraint. We will report that crossover honestly.

Baselines

The naive reference points are linear probing and a standard bottleneck adapter. Linear probing trains only the head on frozen features, and the Dual-Kernel Adapter paper reports it at 86.84% accuracy on COVID at 0.63% data. The same paper reports the standard bottleneck adapter at 83.29% on COVID and 18.95% mIoU on BRATS at 1.25%, and full fine-tuning at 22.39% mIoU on BRATS. LoRA, at 80.43% on COVID, is included as the most common parameter-efficient method. Any adapter that fails to beat linear probing at the smallest data fraction has failed outright.

The strongest existing method is the Dual-Kernel Adapter, which reports 89.01% accuracy on COVID at 0.63% data and 23.02% mIoU on BRATS at 1.25%. It is the strongest competitor because it was designed for exactly this failure and already beats full fine-tuning. We will rerun it with its released settings on identical splits, including the same subsampled training indices across five seeds. It gets the same backbone, the same number of epochs, and the same hyperparameter search budget as our method: a fixed grid over learning rate and adapter width. We will report both its batch-size-1 configuration and the largest batch that fits on a fixed GPU. Our method counts as beating it if it matches its mean accuracy and mIoU within seed noise while using clearly fewer trainable parameters and lower peak memory. Our method counts as clearly better if it exceeds its mean on at least three of the five medical datasets. If it trails by more than seed noise, the claim fails.

Four ablations isolate the key ingredient, all under the same budget. An untied per-layer poly-dilated adapter tests whether tying matters. A tied adapter with a fixed dilation pattern tests whether the shift matters. A tied adapter with a learned layer embedding tests whether the dilation shift beats the conventional way of breaking loop symmetry. A tied Dual-Kernel Adapter tests whether dilation beats large kernels once tying is held constant.

Why not just…?

Why not just keep the Dual-Kernel Adapter and accept the memory cost? Its accuracy is the bar to meet, and if our method only matches it, the contribution is efficiency, not accuracy. The batch-size-1 restriction is a real cost for segmentation, though. It rules out batch normalization statistics and slows training. The peak-memory and maximum-batch measurements, taken on the same GPU, settle whether the savings are material or cosmetic.

Why not just swap the large kernel for dilated $3 \times 3$ convolutions in an ordinary per-layer adapter, with no tying? This is the most serious threat, because it is the simplest reading of the gap and may capture most of the accuracy. If the untied poly-dilated ablation equals the tied version at every data fraction, then tying contributes only a parameter reduction. We would then reframe the paper around dilation alone. We expect tying to help at 0.63–1.25% data through regularization, as the ALBERT results suggest, but only the ablation grid can show it.

Why not just tie the weights and add a learned layer embedding, as looped transformers do? That is the standard fix from On Expressive Power of Looped Transformers, and it costs very few parameters. Our argument is that a learned embedding tells the module its depth but does not change where it looks. The dilation shift does both, and the composition of shifted lattices covers gridding holes. If the learned-embedding variant matches ours on accuracy and receptive-field coverage, we will say so and report the dilation shift as a free but not superior alternative.

Why not just use linear probing, since it already beats standard adapters? It reaches 86.84% on COVID but cannot reshape spatial features for segmentation, and it sits 2.17 points below the Dual-Kernel Adapter even on classification. It stays in the study as the floor.

Proposed Experiments

The main benchmarks are those from the Dual-Kernel Adapter study. For classification we use COVID chest X-rays at 0.63%, 1.25%, and 100%, and ISIC-2019. For segmentation we use BRATS at 1.25% and 100%, BUSI, and ISIC-2018. We report accuracy, mIoU, and Dice, with means and standard deviations over five data-subsampling seeds. For BUSI, ISIC-2018, and ISIC-2019 there is no reported result found on Lacuna, so all numbers there come from our reruns. Efficiency is reported as trainable parameters, peak training memory, maximum batch size, and inference latency. Latency is measured both with standard dilated kernels and with a dilated Winograd implementation, since PSConv warns that non-unit dilation can be slow in standard libraries. As a secondary check that the design is not medical-specific, we run a low-data ADE20K subset against META. META reports 42.2 mIoU for META-T, but only at full data, so only relative trends will be compared.

Three results decide the question. The first is whether the tied shifted-dilation adapter reaches 89.01% on COVID and 23.02% mIoU on BRATS at the scarce fractions with a fraction of the memory. The second is whether the fixed-dilation tied variant shows both lower accuracy and visible gridding holes in its measured receptive field, which would confirm that the shift is the active ingredient. The third is whether the advantage of tying over untying shrinks as the data fraction grows, which is the pattern predicted if tying works by regularization.