Object detectors trained on video-game imagery such as the GTA V-derived Sim10k work well on large, nearby cars in real streets. They fail on small, distant ones. We propose to repair that failure at test time. The detector, trained once on synthetic data and then frozen, extracts features from a real image at several resolutions. A small generative model trained only on synthetic features then "re-renders" those features so they look synthetic again, one resolution level at a time. It starts from the coarse levels, which already transfer well, and uses them as context to repair the fine levels, where small objects live.
The generative model is a flow-matching model: a generator that learns a velocity field which carries random noise along nearly straight paths to data. It needs only a few integration steps and is trained by plain regression, with no adversarial game. The opportunity comes from two findings. Prior work reports that domain transfer in detection is uneven across scale: large objects adapt and small ones do not (Probabilistic Teacher). Separately, tiny objects are recognized mostly through their surroundings rather than their own pixels (Finding Tiny Faces). Together these suggest that the reliable coarse context in a real image can be used to fix the unreliable fine-scale features.
Core Claim
A detector's feature pyramid can be adapted from synthetic to real without paired images and without retraining the detector. The pyramid is the stack of feature maps at strides 4 to 64 produced by a Feature Pyramid Network (FPN). The adaptation is a partial noise-and-regenerate projection through a source-trained flow, run coarse-to-fine, with a separate projection depth for each pyramid level.
The claim is specific and falsifiable. On Sim10k→Cityscapes, this projection should close more of the small-object gap than per-level statistics matching, measured by the difference between small-object and large-object average precision. It should also beat the same flow when run without conditioning on coarser levels. If a per-level affine re-normalization achieves the same small-object gain, the flow is unnecessary and the claim is false.
The closest existing work is FlowEO, which uses conditional flow matching to translate source images into target images inside a single-scale autoencoder latent, then applies frozen task heads. FlowEO's own results constrain our design: it reports that random or pixel-distance optimal-transport pairing fails and that semantically paired images, such as the same location seen by two sensors, are needed. Sim10k has no real counterparts, so that recipe does not apply. We avoid pairing altogether. Our flow is trained only to generate synthetic features from noise, and adaptation uses it as a learned prior to project onto, rather than as a learned source-to-target map.
We take our multi-scale machinery from LapFlow and Pyramidal Flow Matching. Both generate images coarse-to-fine, with finer scales conditioned on coarser ones. Both are purely image and video generators that have never touched detection features or domain shift.
Research Program
We first train a Faster R-CNN with a ResNet-50 FPN backbone on Sim10k. We then freeze it and collect its pyramid features $f_l$ on synthetic images, for levels $l = 2,\dots,6$.
On those features we train a per-level conditional flow $v_\theta^{(l)}(x, t \mid c_{l+1})$. It uses the linear-path conditional flow-matching regression of Flow Matching for Generative Modeling, where the conditioning $c_{l+1}$ is the already-processed coarser level, upsampled. Architecturally this is a lightweight convolutional or windowed-attention network shared across levels with level embeddings. It follows LapFlow's scale-specific-plus-shared design at much smaller size, since feature maps rather than pixels are modeled.
At test time on a real image, we process levels from coarsest to finest. For each level we mix the real feature with fresh noise up to a depth $\tau_l \in [0,1]$ and integrate the source flow forward:
$$x_{\tau_l} = \tau_l f_l + (1-\tau_l)\,\epsilon,\quad \epsilon \sim \mathcal{N}(0,I), \qquad \hat f_l = x_{\tau_l} + \int_{\tau_l}^{1} v_\theta^{(l)}(x_t, t \mid \hat c_{l+1})\,dt .$$
Here $\hat f_l$ is the projected feature passed to the frozen detection heads, and $\hat c_{l+1}$ is the projected coarser level. A depth of $\tau_l = 1$ leaves the level untouched. Smaller $\tau_l$ discards more of the real feature and relies more on the synthetic prior plus coarse context. The noise step matters: a purely deterministic invert-then-regenerate would return the input unchanged, because the ODE is invertible. Straight paths keep the integral cheap, with 2 to 4 Euler steps, as the efficiency literature summarized in Efficient Diffusion Models: A Survey suggests.
The scientific question is how $\tau_l$ should vary with scale. Our hypothesis is that coarse levels need little projection and fine levels need more, mirroring the intra-domain gap. We will report three settings. The first is an oracle that tunes $\tau_l$ on 100 labeled Cityscapes validation images. The second is an unsupervised rule that picks the largest $\tau_l$ at which detections are stable across independent noise draws. The third is a single shared $\tau$ across all levels.
A known risk is that projection hallucinates synthetic-looking cars or erases real small ones. We measure this directly through false-positive and missed-detection counts by size bin. Projected features can also feed pseudo-labels to a single fixed self-training round, in the style of the "AdaBN + Fixed" recipe of Simplifying Source-Free Domain Adaptation. We treat that only as a secondary extension.
The Helmholtz-Hodge split of Velocity Adaptation for Flow-Matching Models separates the velocity field into a structure-carrying conservative part and a texture-carrying rotational part. One could down-weight the rotational part at fine levels to suppress synthetic texture. We list this as an exploratory probe, not a core component, because it requires an extra potential network and a grid search.
Baselines
The naive reference is the frozen Sim10k-trained detector applied directly to Cityscapes. It sets the floor and, by itself, shows how large the small-versus-large gap is. Alongside it we include the same detector trained with more synthetic data, following Driving in the Matrix. That paper reports 0.6803 mAP on KITTI Easy but 0.4207 mAP on KITTI Hard for Sim 200k, and it is the clearest prior evidence that small and occluded objects drive the residual gap.
The cheapest serious competitor is per-level adaptive batch normalization: recomputing feature statistics on unlabeled real images. Simplifying Source-Free Domain Adaptation found that this alone yields a large gain. We extend it to a per-pyramid-level affine whitening-and-recoloring map fit on the same unlabeled Cityscapes images our method sees. This baseline isolates whether anything beyond first- and second-order statistics is needed. To beat it, our method must achieve higher small-object AP at equal or better large-object AP.
The strongest self-training baselines are Cross-Domain Adaptive Teacher, which combines adversarial gradient-reversal alignment with a teacher whose weights track the student by moving average, and SF-UT from the simplification paper. We rerun both on the identical ResNet-50 FPN detector with the same unlabeled target images and a matched GPU-hour budget, using the Sim10k loaders in harsh-99/SCL.
The strongest reported number we found is MTTrans at 57.9 mAP on Sim10k→Cityscapes, using a Deformable DETR detector. Because the architecture differs, we will cite this figure as an external reference point rather than claim a like-for-like comparison. Probabilistic Teacher also evaluates Sim10k→Cityscapes, but no reported result was found on Lacuna for its Sim10k→Cityscapes number. We will rerun it on our detector instead.
Two ablations isolate the key ingredient. The first removes coarse-level conditioning, giving independent per-level flows. The second replaces per-level depths with a single shared depth. If either ablation matches the full method, the coarse-to-fine, scale-dependent story is wrong even if the overall number improves.
Why not just…?
Why not just re-normalize each pyramid level's statistics? This is the most serious threat. AdaBN-style matching is known to be strong (Simplifying Source-Free Domain Adaptation), and it may capture most of the global shift. It cannot use coarse context to change a fine-level feature at a specific location: it applies the same transform everywhere. That spatially varying, context-driven repair is exactly what tiny objects should need (Finding Tiny Faces). The affine baseline is in the main table at matched data. If it equals our small-object AP, we will report that the flow adds nothing beyond moment matching and redirect the project.
Why not just translate synthetic images to look real, then train on them? Image-level translation with flows is the FlowEO route. FlowEO itself shows that semantic pairing is needed for flows to preserve content, and Sim10k has no paired real scenes (FlowEO). Pixel-space translation also spends capacity on appearance details that the detector may ignore. We will include an unpaired, randomly coupled latent flow translation of Sim10k images as a retraining baseline to test whether this concern holds for detection.
Why not just self-train with a teacher on real images? Adaptive Teacher and Probabilistic Teacher report strong numbers on other shifts, for example 50.9 and 47.1 mAP on Cityscapes→Foggy Cityscapes. Their pseudo-labels are weakest exactly on small objects. They also need the source data and repeated retraining, and they risk teacher-student collapse (Simplifying Source-Free Domain Adaptation). Our method is complementary rather than a replacement. The decisive comparison is per-size-bin AP, and the extension experiment tests whether projected features improve a single fixed self-training round over AdaBN-initialized pseudo-labels.
Why not just test on an image pyramid? Upscaling the input helps tiny objects, but it does not remove the domain gap, and it multiplies inference cost up to about 7-fold (Multi-Scale Cross Distillation). We will report three-scale testing of the source detector, with latency, as a compute-matched point of comparison.
Decisive Experiments
The primary benchmark is Sim10k→Cityscapes car detection, with AP50 reported overall and split by box area into small, medium, and large bins. The headline statistic is the small-minus-large AP gap relative to the frozen source detector. The secondary benchmark is Sim10k→KITTI with the Easy and Hard split, chosen because the Hard split isolates small and occluded vehicles and is anchored by the Driving in the Matrix numbers. Every row also reports wall-clock adaptation and inference cost and the number of ODE steps.
Three outcomes would decide the question. The first is whether the full method beats per-level affine matching on small-object AP by a margin exceeding the seed-to-seed spread over three seeds. The second is whether the learned or oracle depths $\tau_l$ fall monotonically from fine to coarse levels, which would confirm that the intra-domain gap is a per-level phenomenon a projection can target. The third is whether removing coarse conditioning erases most of the small-object gain.
A positive result on all three would show that a generative prior over detector features, used coarse-to-fine, is a practical tool for scale-uneven domain shift. A null result on the first, if affine matching suffices, would be equally informative: it would show that the small-object sim-to-real gap is a statistics problem rather than a structural one.
References
- Learning Domain Adaptive Object Detection with Probabilistic Teacher
- Finding Tiny Faces
- Feature Pyramid Networks for Object Detection
- FlowEO: Generative Unsupervised Domain Adaptation for Earth Observation
- LapFlow: Laplacian Multi-scale Flow Matching for Generative Modeling
- Pyramidal Flow Matching for Efficient Video Generative Modeling
- Flow Matching for Generative Modeling
- Efficient Diffusion Models: A Survey
- Simplifying Source-Free Domain Adaptation for Object Detection: Effective Self-Training Strategies and Performance Insights
- Velocity Adaptation for Flow-Matching Models
- Driving in the Matrix: Can virtual worlds replace human-generated annotations for real world tasks?
- Cross-Domain Adaptive Teacher for Object Detection
- MTTrans: Cross-domain Object Detection with Mean Teacher Transformer
- Multi-Scale Cross Distillation for Object Detection in Aerial Images