Object detectors trained on video-game images, such as the GTA V-derived Sim10k car dataset, lose much of their accuracy on real street scenes. The strongest current fix is a teacher-student loop. A "teacher", a slowly updated average of the student's own weights, labels unlabeled real images with soft predictions. The student then learns to match those predictions while it keeps training on synthetic images with exact labels. This note asks a question that loop never asks: which parts of the student should the real-image signal actually change?
Today both signals flow through the same detection head, the final layers that turn features into boxes and class scores. That head is therefore pulled two ways at once: toward rendering-calibrated synthetic labels and toward noisy real-world soft targets. We propose to route the real-image signal around the student's head. On real images, the student's features pass through the teacher's head, and only the features receive the gradient. We use this routing both as a candidate method and as a measuring instrument. It tells us whether the sim-to-real gap in a detector lives mainly in the features or mainly in the head.
Core Claim
CrossKD supplies the method we reuse. In ordinary single-dataset distillation, it found that making a student's head match both ground truth and a teacher's predictions produces "target conflict," meaning the two objectives push the same parameters in contradictory directions. It fixed this by feeding the student's intermediate head features into the frozen teacher head, reporting +3.5 AP for GFL on COCO (40.2 to 43.7).
ALDI++ motivates the proposal. It is the strongest fair method for synthetic-to-real detection, and it distills soft outputs from all heads directly into the student's own heads. That is exactly the setup in which CrossKD's conflict should be worse, because the two targets now come from different visual worlds.
We claim three things. First, during synthetic-to-real adaptation, the gradient from synthetic labels and the gradient from real-image distillation are measurably opposed, and the opposition is concentrated in the head layers rather than the backbone. Second, routing the real-image loss through the teacher's head reduces that opposition. Third, the reduced opposition should yield a gain over ALDI++ that is small but repeatable.
There is an open tension the work must settle. DA-Pro argues the opposite of our premise: the head itself is domain-biased and must be adapted, which it does with vision-language prompts. Simplifying SFOD similarly finds that fine-tuning the box-regression head on the target helps. If they are right, keeping real-image gradient out of the head will hurt. The experiment is designed so that either outcome is informative.
Research Program
All work lives inside the ALDI codebase. We keep ALDI++'s burn-in recipe of strong augmentation plus an exponential-moving-average teacher, and its soft multi-task distillation. We change only where the real-image distillation loss sends its gradient.
The central object is a family of routings indexed by a cut depth $k$ inside the detection head. Real-image features computed by the student up to head layer $k$ are passed through the teacher's head layers above $k$. The loss compares that output to the teacher's own prediction on the same image. The student's own head layers above $k$ receive gradient only from synthetic ground truth. Setting $k$ at the head input recovers a full CrossKD-style route. Setting $k$ at the head output recovers ALDI++. Intermediate values interpolate.
We pair this family with the opposite control, in which the real-image loss updates only the head and leaves the backbone fixed. This control directly operationalizes DA-Pro's "the head is the problem" view, but without needing CLIP.
Before any accuracy run, we measure conflict directly. At checkpoints during ALDI++ training, we compute the cosine similarity between the synthetic-label gradient and the real-distillation gradient, separately for each backbone stage and each head layer. Negative values mean the two losses pull the same weights in opposite directions. This profile is the first deliverable. It predicts which cut depth should help, and it can falsify the premise before compute is spent on method tuning. If the profile shows no opposition in the head, we report that and stop the method line.
There is one subtlety. Because the teacher is an average of past students, its head is close to the student's head. Cross-head routing therefore differs from simply detaching the student's head parameters for the real-image loss in only two ways: the teacher head lags behind, and it is not updated by synthetic labels within the current step. We treat that detached variant as the key ablation rather than hiding it.
We also test a frozen source-only teacher head, which never sees real images, in place of the averaged teacher. This answers the question the research file flags as untested: can a synthetic-trained head serve as a reliable "interpreter" for real features?
Baselines
Source-only reference. The naive reference is ALDI's tuned source-only detector, trained on Sim10k with the same augmentations and schedule and no target data. ALDI showed this baseline beats many published adaptation methods. Every gain we report is stated relative to it.
ALDI++. The strongest competitor is ALDI++ itself. It is the only method the file reports as beating a fair baseline on Sim10k→Cityscapes: +5.7 AP50, and within 1.0 AP of a model trained on labeled Cityscapes. It is run from the released configuration with the same backbone, iterations, batch composition and seeds as our variants. Because our method is a one-line change in gradient routing inside that configuration, compute and data are identical by construction. Any hyperparameter we tune (the cut depth, and a loss weight) gets a matching tuning budget spent on ALDI++'s distillation weight. Beating it means a mean AP50 improvement over at least three seeds that exceeds the seed standard deviation.
Gradient-allocation ablations. Three ablations isolate the key ingredient. The detached-head variant shows whether the teacher head matters or only gradient allocation does. The head-only variant tests whether adapting the head is what helps. The frozen-source-teacher-head variant tests whether a synthetic-trained head is a usable interpreter.
Context numbers, not baselines. Published figures from other systems are listed for context only, since they use different architectures and are not re-run: MTTrans reports 57.9 mAP, SIGMA++ 53.7 mAP, and DA-Pro 62.9 mAP with a CLIP backbone, all on Sim10k→Cityscapes.
Why not just…?
Why not just stop the real-image gradient from reaching the student's head, with no teacher-head routing? This is the most serious threat. It may capture most of the effect, and we test it as a first-class arm. If it matches cross-head routing, we report the result as "the gain comes from gradient allocation, not from the teacher head." That is a simpler and more useful recipe, and we would recommend it over the more elaborate method.
Why not just add Fourier-amplitude augmentation? PASTA adds high-frequency amplitude noise that synthetic images lack, reporting +16.9 mAP over its baseline on Sim10k→Cityscapes. Augmentation, however, changes what the inputs look like. It does not change which parameters two opposing losses fight over. ALDI++ already uses strong augmentation during burn-in, and it still leaves the head shared. We add PASTA to both ALDI++ and our variant. If the gap between them disappears, conflict was an artifact of weak augmentation.
Why not just adapt the head, as DA-Pro does? DA-Pro's evidence is real, but it is tied to CLIP text prompts and does not transfer to plain CNN or ViT detectors. The head-only arm is our unimodal stand-in. If the head-only arm wins and feature routing loses, the sim-to-real gap lives in the head, and our premise is wrong.
Why not just recalibrate normalization statistics? Recalibrating batch-normalization statistics on the real domain helps, as shown by Simplifying SFOD. It is unavailable for the layer-normalized transformer and ConvNeXt detectors ALDI supports, and that paper still needed self-training on top of it. We include it as a starting point for the CNN backbone only.
Proposed Experiments
The primary benchmark is Sim10k→Cityscapes car detection, measured by AP50. We must be candid about headroom: ALDI++ is reported within 1.0 AP of the labeled-target oracle there. Our success threshold is therefore closing roughly half that remaining gap, consistently across seeds, and not a large jump.
Because headroom is so small there, the gradient-conflict profile and the cut-depth sweep carry most of the scientific weight on Sim10k. We add CFC-DAOD sonar fish detection, where ALDI++ gains only +0.6 AP50. A non-RGB, low-signal shift there may show larger differences between routings. We also add Cityscapes→Foggy Cityscapes as a control, since it is a weather shift rather than a rendering shift. If the method helps there equally, the effect is about generic domain shift rather than sim-to-real specifically.
Each routing is run on a ResNet-50 and a ConvNeXt backbone, both supported by ALDI. This checks that conclusions do not depend on the normalization scheme.
Three outcomes decide the work. The first is whether head-layer gradient opposition exists and shrinks under routing. The second is whether feature routing beats ALDI++, and also the detached-head control, beyond seed noise on at least two of the three shifts. The third is whether the head-only arm instead wins, which would support DA-Pro's account.
We will also report the per-class and per-object-size breakdown of where the AP changes. This shows whether routing mainly helps hard, small, or occluded cars, where synthetic labels are least realistic.
References
- CrossKD: Cross-Head Knowledge Distillation for Object Detection
- Align and Distill: Unifying and Improving Domain Adaptive Object Detection
- Learning Domain-Aware Detection Head with Prompt Tuning
- Simplifying Source-Free Domain Adaptation for Object Detection: Effective Self-Training Strategies and Performance Insights
- MTTrans: Cross-domain Object Detection with Mean Teacher Transformer
- SIGMA++: Improved Semantic-Complete Graph Matching for Domain Adaptive Object Detection
- Pasta: Proportional Amplitude Spectrum Training Augmentation for Syn-to-Real Domain Generalization