Letting Auxiliary Pyramid Heads Choose the Fusion Kernels

Detectors and segmenters build a feature pyramid, a stack of feature maps at several resolutions in which coarse, semantically rich maps are upsampled and merged into fine, spatially precise ones. The convolutions that perform this merging are static: every image is fused with the same filters, whether it shows one large object or a hundred small ones. A separate line of work makes convolution weights input-dependent by mixing a small bank of kernels with coefficients predicted per image. A third line attaches auxiliary prediction heads to intermediate layers, a technique called deep supervision. Until now these three ideas have been combined only by stacking them. Dynamic kernels go in the backbone, auxiliary losses go on the side, and the two never talk to each other.

This proposal connects them through one mechanism. At each pyramid level, a cheap auxiliary head predicts what is present at that scale: a coarse, softened map of which classes appear. That prediction, and not an unsupervised pooled feature vector, is the input to the small controller that decides how to mix the fusion kernels for that level. The auxiliary loss therefore stops being a passive extra gradient and becomes a supervised, content-aware signal that steers weight selection. We expect this to fix two known weaknesses at once. Dynamic convolution's routing tends to collapse onto the same kernels for every input, and deep supervision has no forward influence on how the network computes.

Core Claim

In a feature pyramid decoder, kernel-mixing coefficients computed from scale-specific auxiliary predictions trained with soft, schedule-calibrated targets will give more diverse routing and better dense-prediction accuracy than either of two alternatives at matched parameters and compute. The first alternative is the same dynamic kernels driven by the usual global-pooling controller, even when the same auxiliary losses are added alongside. The second is feature-level attention driven by the same auxiliary predictions. Concretely, for pyramid level $l$ with fused input $x_l$, auxiliary prediction $\hat y_l$, and a kernel bank $\{W_1,\dots,W_K\}$, the fusion convolution uses

$$W_l(x) = \sum_{i=1}^{K} \alpha_i\big(g(\hat y_l),\, e_l\big)\, W_i ,$$

where $g$ pools the auxiliary class map into a short descriptor, $e_l$ is a learned level embedding, and $\alpha$ is a softmax over the $K$ kernels. Training minimizes

$$L = L_{\text{task}} + \sum_{l} \lambda_l \, \mathrm{CE}\big(\hat y_l,\ \tilde y_l(\tau_l)\big),$$

where $\tilde y_l(\tau_l)$ is the downsampled ground truth softened with temperature $\tau_l$, and $\lambda_l$ weights each level's auxiliary loss.

What is new is the direction of the arrow. Supervision at each scale decides which weights that scale uses. No paper in our search does this.

Relation to Prior Work

The kernel-mixing operator itself is borrowed. CondConv and Dynamic Convolution mix expert kernels using coefficients from global pooling plus a small fully connected network. ODConv extends the attention to spatial, channel and filter dimensions. KernelWarehouse shares kernel pieces across neighbouring layers to control parameter count. FDConv partitions weights in the Fourier domain so that the experts do not learn redundant frequency responses. All of them place dynamic kernels in backbones and drive them with unguided controllers. The research file records that these controllers can collapse and need temperature annealing or sum-to-one constraints. That collapse is the problem we target. We keep their operators and change what the controller sees and where the operator is placed.

The closest pyramid work motivates the proposal but leaves the gap open. DyFPN gates each lateral connection between a heavy Inception block and a light 1×1 bypass. That is a binary, module-level choice trained only by the task loss and a compute penalty. DetectoRS blends two dilation rates of a single shared kernel through a pixel-wise switch. DPNet routes among fixed kernel sizes in a salient-object decoder. CCCUnet uses dynamic kernels only in a U-Net encoder, with static fusion and a terminal-only loss. None of these modulate the kernel weights of the fusion convolutions, and none use intermediate supervision to steer adaptation.

Two papers constrain the design. Scale-Equalizing Pyramid Convolution shows that deep pyramids are not true scale-spaces, so kernels shared across levels are misaligned. This is why our controller receives a level embedding, and why we test sharing a single kernel bank across levels as an explicit arm rather than assuming it works. Deep Trajectory Supervision shows that hard targets on immature intermediate layers create an "impedance mismatch," and that softening early targets on an exponential schedule beats greedy deep supervision. It also supplies our temperature schedule $\tau_l$. That evidence was gathered on feedforward classifiers, though, and in a pyramid it is unclear which level counts as "early." The coarsest map is computed first in the top-down pass, but the finest map rests on the shallowest backbone features. We treat that ordering as an empirical question. Deeply-Supervised Nets is the original greedy companion-loss recipe and serves as our naive deep-supervision arm.

Research Program

We build a pyramid decoder whose lateral 1×1 and smoothing 3×3 convolutions are replaced by kernel-mixing convolutions. We implement two variants: plain scalar mixing as in Dynamic Convolution, and FDConv's frequency-disjoint weights, which file evidence suggests add parameters most frugally. A 1×1 auxiliary classifier on each pyramid level produces $\hat y_l$. Its class-presence descriptor, together with the level embedding, feeds a two-layer controller.

Three design questions are studied in order. The first is whether gradients from the controller should flow back into the auxiliary head or be detached. Detaching forces the head to be a pure content predictor, while attaching lets the router reshape it. The second is how the soft-target schedule should be ordered across levels: coarse-to-fine, fine-to-coarse, or uniform. Linear probes on each level, in the spirit of Deep Trajectory Supervision's tuned-lens analysis, will tell us how semantic evidence actually accumulates through the top-down pass before we fix the schedule. The third is whether one kernel bank can be shared across all levels, with only the coefficients differing by level, without the misalignment SEPC reports.

Alongside accuracy, we log routing diagnostics: the entropy of the mixing coefficients averaged over images, their variance across images, and their mutual information with the image's dominant object scale. These diagnostics test the mechanism directly, not just the outcome.

The main testbed is semantic segmentation on ADE20K with a ResNet-50 Semantic-FPN-style decoder, which has no built-in per-level supervision. The second is COCO detection with Faster R-CNN plus FPN on ResNet-50 (COCO). Detection already supervises every level through anchor assignment, so the auxiliary head there predicts a softened class-presence map rather than adding a new detection loss. This makes COCO a harder test of whether the conditioning, and not the extra supervision, carries the gain. A third, smaller check is WHU-CD building change detection, where the strongest file entry, CCCUnet, uses a terminal-only loss.

Baselines

The naive reference is the unchanged static pyramid decoder with terminal loss only, trained on the same schedule. Every other arm is reported as a difference from it. The second reference adds per-level auxiliary heads trained with hard targets, following Deeply-Supervised Nets. The third swaps in soft scheduled targets, following Deep Trajectory Supervision. Together these show how much deep supervision alone contributes to a static pyramid.

The strongest existing dynamic-weight competitor is FDConv. On ADE20K, the file reports 43.8 mIoU for FDConv against 43.5 for KernelWarehouse. On COCO it reports 39.4 box AP with a ResNet-50, against 39.2 for ODConv. Those figures come from inserting dynamic convolution into the backbone, and the file does not state the decoder used. We therefore rerun FDConv-in-backbone in our own pipeline rather than quoting it. We also run FDConv placed in the pyramid fusion convolutions with its native pooling controller.

For pyramid-level adaptivity on COCO, we compare against DyFPN, which the file reports cuts lateral FLOPs from 896.2G to 537.5G at a cost of 0.2 AP relative to Inception FPN. We also compare against SEPC, reported at roughly a 2 AP boost for two-stage detectors. On WHU-CD the reference is CCCUnet at 93.86 F1 and 80.97 IoU. For saliency-style decoders there is a DPNet result in the file, but only as a relative F-measure gain over CTDNet, with no absolute baseline we could reuse.

The ablation that isolates our key ingredient keeps everything identical, including the dynamic fusion kernels, the auxiliary heads and the soft-target losses. It changes only the controller's input, swapping the auxiliary-prediction descriptor for the standard global-pooled feature. A second ablation keeps the auxiliary-conditioned controller but applies its output as channel attention on features with static kernels. A third replaces the dynamic arms with a statically widened pyramid matched in parameters and FLOPs.

Fairness rules apply to all arms. Each uses the same backbone initialization, the same training schedule and augmentations, and three seeds. Every arm with auxiliary losses receives an identical grid search over the loss weight and schedule curvature. FLOPs, parameters and measured GPU latency are reported for every arm, including kernel-assembly time, which the file flags as a hidden cost of dynamic convolution. Beating the strongest competitor requires two things. The proposal must exceed the GAP-controller-plus-auxiliary ablation by more than the seed-to-seed standard deviation on ADE20K, and it must match or exceed rerun FDConv at no more parameters.

Why not just…?

Why not just add auxiliary losses to an FDConv- or ODConv-equipped pyramid and leave the controller alone? This is the most serious threat, because the combination is trivial to implement and might capture all of the benefit. Gradients from the auxiliary heads already reach the controller's inputs indirectly. Our isolating ablation is exactly this configuration. If it matches the proposal on accuracy and routing entropy, the claim is false and the paper reduces to a negative result about controller inputs. We commit to reporting that outcome.

Why not just use the auxiliary predictions to drive feature attention, as squeeze-and-excitation does, and keep kernels static? Dynamic Convolution and ODConv report that modulating kernels beats modulating outputs, but they did so with unguided controllers in backbones, not with supervised signals in a decoder. The feature-attention arm uses the same conditioning signal, so it separates "better signal" from "kernel-level modulation." If it ties the proposal, the contribution is the signal rather than dynamic weights, and we would reframe accordingly.

Why not just put dynamic convolution in the backbone, where it is already known to help? FDConv's dense-prediction gains come from backbone placement. Our rerun of that configuration, plus a backbone-and-pyramid combination, shows whether fusion-stage adaptation adds anything. The pyramid is where scale-specific content differs most, which is the reason to expect it does.

Why not just widen the static pyramid? File evidence from KernelWarehouse and Dynamic Convolution shows dynamic kernels outperforming static capacity at matched FLOPs in classification. Whether this holds in decoders is unestablished, so the parameter- and FLOP-matched wide pyramid stays in every table.

Decisive Checks

The idea stands or falls on two measurements in the ADE20K setting. The first is accuracy: the auxiliary-conditioned controller must beat the GAP controller with identical auxiliary losses. The second is mechanism: its mixing coefficients must show higher across-image variance and measurable dependence on object scale, while the GAP controller's coefficients collapse toward a constant. Accuracy gains without the routing signature would suggest the effect comes from something other than the claimed mechanism. The routing signature without accuracy gains would show that diverse routing is not what limits these decoders.

Secondary checks address the remaining open questions. One is whether the gain survives on COCO, where per-level supervision already exists. Another is which ordering of soft-target temperatures across levels the probes support. The last is whether a single kernel bank shared across levels, with level-conditioned coefficients, recovers SEPC-like gains without 3D convolution, or instead shows the misalignment SEPC predicts.