Primary-Anchored Gradients for Detector Scaffolding Heads

Many modern object detectors add extra heads during training that are deleted before deployment. A transformer detector might get a dense auxiliary head or a second decoder that gives it many more training targets; at test time only the original head runs, so speed is unchanged. Every such design adds its auxiliary loss to the main loss with a fixed weight chosen by hand. That weight is tuned for one detector, one dataset and one labeling regime, and nobody knows whether it carries over. This proposal tests a simple change. Measure the gradient that each loss sends into the feature map the heads share. At each spatial location, remove the part of the auxiliary gradient that points against the main head's gradient, and keep everything else. The discarded head is scaffolding and should never be allowed to pull the deployed head backwards, but it should not be switched off either.

The claim is about hyperparameter sensitivity across architectures. We predict that this per-location, main-head-anchored projection makes accuracy nearly flat in the auxiliary weight. If so, one untuned setting would match the per-architecture tuned optimum on transformer, single-stage and two-stage detectors, and under domain shift and missing labels, where hand-tuned weights are most likely to break. If a tuned or annealed static weight does just as well everywhere, that is also a useful finding: balancing would not matter for scaffolding, and the field could stop worrying about it.

Why this gap is real

Two recent transformer detectors show that training-only heads are worth having. RT-DETRv3 adds three training-only components to a real-time transformer detector (DETR-style, meaning each ground-truth object is matched to exactly one learned query). The first is a convolutional head on the encoder output with one-to-many label assignment. The second perturbs self-attention with random masks. The third is a shared-weight decoder branch with one-to-many matching. All three are deleted at inference.

DAC-DETR observes that cross-attention gathers queries around objects while self-attention spreads them apart. It adds a training-only decoder without self-attention that shares weights with the main decoder. Both papers motivate this proposal, because they establish that dense auxiliary supervision helps. Both also leave the loss combination as a static sum.

Gradient-balancing work supplies the method:

None of these has been applied to training-only detector heads. Two features of detection make them a poor fit as they stand.

The first is asymmetry. Only one head ships. A Pareto compromise in the style of the multi-objective and conflict-averse methods would trade main-head quality for an auxiliary head that will be deleted.

The second is spatial structure. A detector's shared feature map is mostly background. A single global cosine over all parameters is dominated by background locations, and Conflict-Alleviated Gradient Descent reports that detection gradients conflict frequently. Together these suggest a global kill switch would discard the auxiliary signal for long stretches, including at locations where it agrees with the main head.

Core Claim

Let $z$ be the shared feature map that both heads read: the encoder output in a transformer detector, the feature pyramid in a single- or two-stage detector. Let $z_u$ be its feature vector at spatial location $u$. Let $p_u = \partial L_{main}/\partial z_u$ and $a_u = \partial L_{aux}/\partial z_u$ be the gradients from the deployed head's loss and the scaffolding loss at that location. We replace $a_u$ by

$$ \tilde a_u = a_u - \frac{\min(0, \langle a_u, p_u\rangle)}{\|p_u\|^2 + \epsilon}\, p_u , $$

which leaves $a_u$ unchanged when it agrees with the main gradient. When it disagrees, the formula removes only the opposing component; $\epsilon$ is a small constant for numerical stability. The backbone then receives the upstream gradient $p_u + \lambda \tilde a_u$, where $\lambda$ is the auxiliary weight.

This costs two backward passes through the lightweight heads, as far as the feature map. It needs one backward pass through the backbone, which is the same cost-saving logic as the multi-objective paper's shared-representation trick. Parameters that belong only to the auxiliary head receive their raw gradient, since they do not affect the deployed model.

The testable prediction has two parts. With projection, final main-head average precision (AP) varies little across $\lambda \in \{0.25, 0.5, 1, 2, 4\}$. And the default $\lambda=1$ is within noise of each architecture's tuned static optimum, with no per-architecture search.

Research Program

The work has three stages: a diagnostic, the method, and a portability study.

Diagnostic. Before changing training, we measure where conflict happens. Along the training trajectories of the published scaffolded detectors, we log the per-location cosine between $a_u$ and $p_u$ and the magnitude ratio $\|a_u\|/\|p_u\|$. These are split by pyramid level and by region: matched-object locations, near-duplicate locations (positive under one-to-many but background under one-to-one), and pure background.

This diagnostic carries a real risk for the method. Near-duplicate locations are where the dense head deliberately disagrees with the sparse head, and that disagreement may be exactly the useful extra supervision. If the diagnostic shows the opposing component is concentrated there, projection could erase the benefit. The ablations below are built to detect that. The diagnostic is also a contribution by itself, because neither scaffolding paper reports how its heads interact at the shared features.

Method. We implement the projection as a gradient hook at $z$ and apply it to the native scaffolds of RT-DETRv3 and DAC-DETR.

In RT-DETRv3 the auxiliary decoder shares weights with the main decoder, so those shared decoder weights receive mixed gradients. Projection at the encoder output does not touch them. We treat projecting the shared-decoder parameter gradients as a separate ablation rather than the default, because DAC-DETR's analysis suggests the decoder-level disagreement is the intended mechanism.

Portability. To test transfer across detector families, we attach one scaffold that does not depend on architecture to three detectors: FCOS (single-stage), Faster R-CNN (two-stage) and RT-DETR (transformer). The scaffold is a BatchFormer-style transformer that runs across the batch dimension of pooled features, with the shared-classifier trick that lets it be deleted after training. BatchFormer was shown only on classification backbones, so this tests whether a portable scaffold needs per-architecture weight tuning, and whether projection removes that need.

Baselines

No scaffold. The naive reference is each detector trained without auxiliary heads under the same schedule. For RT-DETR-R18 on COCO val2017, RT-DETRv3 reports that its scaffolding reaches 48.1% AP, a gain of 1.6 over RT-DETR-R18. For Deformable DETR R50 at 12 epochs, DAC-DETR reports 43.7 AP without and 47.1 AP with its auxiliary decoder.

Published static weights. This is the strongest existing method and the number to match: the published static weights of RT-DETRv3 and DAC-DETR, rerun in our codebase. We will report our reproduced numbers next to the cited 48.1 and 47.1 so readers can see any reproduction gap.

Tuned static weight. The real competitor is a per-architecture grid search over $\lambda$ on the same five values. Fairness here means charging the grid search its full five training runs, while the projection method gets one run at $\lambda=1$.

Annealed weight. A static $\lambda$ decayed linearly to zero over training, tuned the same way.

Existing balancing methods. Adapting Auxiliary Losses Using Gradient Similarity, in both its global-cosine gating and scaling forms. Symmetric CAGrad from Conflict-Averse Gradient Descent, applied at the same feature map with its default conflict-aversion radius. And MGDA-UB from Multi-Task Learning as Multi-Objective Optimization.

Ablations. These isolate the key ingredients:

  • global projection, which isolates the effect of being per-location;
  • per-location magnitude normalization without projection, which isolates direction versus scale;
  • projection restricted to background locations only, which tests whether removing near-duplicate disagreement hurts.

All runs use identical data, epochs, augmentation and three seeds. Beating the competition means one of two outcomes. Either projection at $\lambda=1$ is within one seed standard deviation of the tuned static optimum on every architecture and setting while using one-fifth of the tuning compute. Or it exceeds the tuned optimum where the tuned weight is transferred from another setting.

Why not just…?

Why not just grid-search one scalar weight? With a single auxiliary loss this costs five runs, and it is a genuine threat. Our answer rests on a transfer experiment, not on cost alone. We tune $\lambda$ on COCO and then reuse it on Cityscapes, on COCO-miss50p and on a different detector family. If the transferred weight stays within noise of each setting's own optimum, tuning is cheap and portable, and our method's case collapses to a modest compute saving. We will report that outcome plainly. The uniform-scaling penalty on CelebA in Multi-Task Learning as Multi-Objective Optimization (9.62% versus 8.25% error) suggests static weights can be badly off, but that is a multi-task classification result, not evidence about detectors.

Why not just anneal the auxiliary weight to zero? Because the head is deleted, fading it out late in training is a natural fix for any end-of-training harm. It cannot prevent early conflict from shaping the backbone, but it might be enough in practice. The annealed baseline is tuned with the same budget, so the comparison settles this directly.

Why not just gate with a global cosine? Adapting Auxiliary Losses Using Gradient Similarity does exactly that and works in classification. Detection gradients conflict often, per Conflict-Alleviated Gradient Descent, so a global gate may switch off agreeing object locations along with conflicting background. The global-projection ablation and our logged gate-off fraction measure this. If global gating matches per-location projection, we would drop the spatial claim.

Why not use symmetric CAGrad or MGDA? These methods seek a compromise that also protects the auxiliary loss. Because the auxiliary head is deleted, protecting its loss could cost the deployed head accuracy. Comparing them against projection on main-head AP, under the same training budget, measures whether that concern matters in practice.

Proposed Experiments and Decisive Checks

COCO val2017 sensitivity sweep. This is the main experiment. Run the five-value $\lambda$ sweep with and without projection on RT-DETR-R18 with RT-DETRv3 scaffolding, and on Deformable DETR R50 at 12 epochs with the DAC-DETR auxiliary decoder.

Portability. Run the same sweep with the BatchFormer-style scaffold on FCOS, Faster R-CNN and RT-DETR.

Domain shift. Train on Cityscapes and evaluate on both Cityscapes and Foggy Cityscapes without adaptation. This tests whether projection protects robustness that a scaffold could erode. For scale, Conflict-Alleviated Gradient Descent reports 43.5% mAP for an adapted convolutional detector. We found no Lacuna result for transformer detectors with training-only scaffolds on Foggy Cityscapes, so that comparison is ours to establish.

Missing labels. On COCO-miss50p, one-to-many assignment labels unannotated objects as background in dense neighborhoods, so conflict should be sharper there. The reference is Co-Student at 37.0 AP with FCOS.

Primary metric. The spread of main-head AP across the $\lambda$ sweep, with and without projection.

Secondary metrics. AP and AP for small objects at $\lambda=1$ versus the tuned optimum, the tuning compute spent, and wall-clock training overhead. Inference latency should be identical by construction, and we verify it.

Decisive check. The weight-transfer matrix described under the grid-search question above. Rows are the setting where $\lambda$ was tuned; columns are the setting where it is applied. We compare against projection at $\lambda=1$.

The proposal succeeds if projection's AP spread across $\lambda$ is at most half that of static weighting, and it matches or beats every transferred static weight. It fails if the annealed or tuned static baselines match it everywhere. It also fails, informatively, if the background-only ablation beats full projection, which would mean the opposing gradients near objects are the useful part of dense supervision.