Test-Time Channel Reliability for Synthetic-to-Real Transfer

Networks trained on rendered images, such as street scenes from a video game, lose a large share of their accuracy on real camera images. One known reason is that renderers produce images that are too smooth: real photographs carry much more fine, high-frequency texture variation than synthetic ones. A network trained only on smooth renders can come to rely on internal features that respond to rendering-specific texture. On a real image those features fire unpredictably. We propose a simple test-time rule that finds such features on each real image and turns them down. For each image, the method applies a few cheap perturbations to the image's frequency spectrum, of the kind the synthetic-to-real gap consists of. It measures how much each internal feature channel (one of the many parallel activation maps in a convolutional layer) changes, and down-weights the channels that swing the most. The rule needs no real training data, no extra parameters and no gradient updates.

The idea transplants a mechanism from classical visual tracking into deep sim-to-real models. In tracking, a feature's reliability is judged on the spot from how its response behaves on the current frame. Here, the reliability signal is designed to target the specific nuisance that separates simulators from cameras. The question we test is narrow. Does per-image, perturbation-derived channel weighting recover real-domain accuracy beyond what one gets by spending the same extra forward passes on ordinary test-time augmentation? And does it add anything once the model was already trained with frequency augmentation?

Core Claim

A channel's sensitivity to amplitude-spectrum perturbation of the current test image is a usable, label-free estimate of whether that channel is relying on synthetic-specific appearance. We expect channels with high sensitivity on real images to be the ones whose removal improves real-domain predictions. Weighting channels by an image-specific stability score at inference should therefore improve synthetic-to-real generalization without access to target data.

Concretely, let $a_c(x)$ be the activation map of channel $c$ at a chosen backbone stage for image $x$. Let $T_1,\dots,T_K$ be random amplitude perturbations that keep the phase and jitter high-frequency amplitudes more strongly than low ones. Let $\bar a_c(x)$ be the mean of $a_c(T_k x)$ over $k$. The instability score is

$$s_c(x) = \frac{\frac{1}{K}\sum_{k=1}^{K} \lVert a_c(T_k x) - \bar a_c(x) \rVert^2}{\lVert \bar a_c(x) \rVert^2 + \epsilon},$$

and the channel is rescaled by

$$w_c(x) = \frac{\exp(-s_c(x)/\tau)}{\frac{1}{C}\sum_{c'} \exp(-s_{c'}(x)/\tau)}.$$

Here $\epsilon$ is a small constant, $C$ is the number of channels, and the temperature $\tau$ controls how sharply unstable channels are suppressed. Normalizing by the mean keeps the stage's overall activation scale unchanged. The forward pass then continues on the unperturbed image with $w_c(x)\,a_c(x)$. Because nothing is learned, the gate cannot collapse to uniform or zero weights during training. It also cannot inherit source-domain shortcuts the way a learned gate would. The research file identifies both failure modes as open problems for neural gating.

Relation to Prior Work

Two tracking papers supply the mechanism. CSR-DCF down-weights feature channels whose correlation responses are ambiguous on the current frame. TACF recomputes channel weights per frame from response sharpness. Both operate on hand-crafted features and judge reliability by response shape. We keep their "judge reliability from the current input" principle. We replace response shape with stability under a perturbation family chosen to mimic the sim-to-real gap, and we apply it inside deep segmentation and detection backbones.

PASTA motivates the perturbation family and is also our strongest competitor. It shows synthetic images lack high-frequency variance, and it perturbs input amplitude spectra during training in proportion to frequency. PASTA uses perturbations to change what the network learns. We use the same kind of perturbation at test time as a probe that decides, image by image, which learned channels to trust. The two are compatible, and whether they are complementary is a central experiment.

Three domain-generalization methods constrain the design because they already handle synthetic-to-real without target data. Semantic-Aware Domain Generalized Segmentation normalizes and whitens channel groups defined by classifier weights. Its grouping is fixed per category, not driven by per-image reliability. Automated Synthetic-to-Real Generalization prevents forgetting of ImageNet features with a KL constraint to a frozen pre-trained network and reinforcement-learned learning rates per layer. Its weighting is coarse, one rate per block, and acts only during training.

Two methods that weight channels across domains need unlabeled real data. BIWAA uses classification gradients as a channel attention map to steer adversarial alignment. Transfer Channel Pruning permanently removes channels that inflate the mean discrepancy between source and target feature distributions. TCP is the closest in spirit: it supports the premise that some channels within a layer cause negative transfer. It differs from our method in requiring target samples and in being static and hard rather than per-image and soft.

Quantifying and Bridging the Fidelity Gap provides both motivation and a diagnostic. It shows that matching outputs can widen the gap in the internal "decisive features" a model uses, by 16% on drivable-area segmentation. We borrow its decisive-feature distance as a mechanism-level check of whether our gating moves real-image reasoning closer to synthetic-image reasoning.

Research Program

The work has three parts.

The first part is a falsifiable diagnostic run before any method claims. We train source-only and PASTA-trained segmentation models on GTA5. On held-out labeled Cityscapes validation images, we compute two quantities per channel. One is the instability score $s_c$. The other is a "harm" score: the change in per-image mIoU when that channel is zeroed. Labels are used only for this analysis, never by the method. The hypothesis survives if instability correlates positively with harm on real images. It should correlate more strongly on real images than on synthetic validation images. If it does not, the project stops or pivots to a different probe.

The second part implements the gate at two or three backbone stages of DeepLabv3+ and ResNet-50 segmentation models, and of a Faster R-CNN detector. We study $K$ (the number of perturbations), the stage choice and $\tau$. All three are selected only on synthetic validation data, so the method stays target-free. We report sensitivity curves so that readers can see how much target-free selection costs.

The third part is mechanism analysis. We visualize which channels get suppressed and whether they concentrate on texture-heavy classes such as road, vegetation and building facades. We also measure decisive-feature distance on matched KITTI and VirtualKITTI2 pairs with and without gating.

Baselines

The naive reference is the source-only model trained on GTA5 and evaluated directly on real data. For DeepLabv3+ on GTAV→Cityscapes, PASTA reports 28.95 mIoU. We will retrain it under our exact recipe rather than rely on that figure.

The strongest existing method is PASTA itself. It reports 44.12 mIoU on GTAV→Cityscapes with DeepLabv3+, and a +16.9 mAP gain over its baseline on Sim10k→Cityscapes detection; the file gives no absolute detection number. We will reproduce PASTA with identical backbone, schedule and data. Gating will be applied both to a source-only model and to the PASTA-trained model. Beating PASTA means a statistically reliable gain over three or more seeds when gating is added on top of PASTA training, not merely gating a weaker model.

Semantic-Aware Domain Generalized Segmentation is the strongest feature-level competitor. It reports 39.75 mIoU with ResNet-50 against its 29.32 baseline. We run it on the same ResNet-50 setup and also test gating stacked on it. Automated Synthetic-to-Real Generalization is included for VisDA-17 classification, where it reports 61.1% accuracy. It also reports 29.64 mIoU with FCN-Res50 on GTA5→Cityscapes.

Two ablations isolate the key ingredient. The first is matched test-time augmentation. It uses the same $K$ perturbed images, but averages final predictions instead of computing channel weights. This holds compute fixed and asks whether the gating, rather than the extra forward passes, does the work. The second is a static mask. It computes instability scores once, averaged over synthetic validation images, and applies a fixed channel weighting to every test image. This asks whether per-image adaptivity matters or a global "untrustworthy channel" list suffices. A third control replaces the frequency-proportional perturbations with plain color jitter. This tests whether the probe's specificity to the sim-to-real gap matters.

Why not just…?

Why not just train with PASTA and stop there? This is the most serious threat. PASTA-trained networks may already be stable under exactly these perturbations, leaving the probe nothing to find. If gating on top of PASTA yields no gain over seeds, we will report the method as subsumed by training-time augmentation. The diagnostic in part one already tests this: it checks whether instability still correlates with harm in PASTA-trained models. The case for a residual gain is that augmentation teaches invariance on average over synthetic content. Individual real images, such as night scenes, glare or unusual textures, can still activate unstable channels that no training-time schedule anticipated.

Why not just average predictions over the perturbed images? Test-time augmentation spends the same compute and is the obvious way to use perturbations at inference. It smooths outputs but still lets unstable channels contaminate every forward pass. Gating instead removes their contribution before it reaches later layers. The matched-compute test-time augmentation ablation decides this directly. If averaging matches gating, the channel-level mechanism is unnecessary and we will say so.

Why not just recompute normalization statistics or adapt the network on real test batches? Batch-statistic recalibration and entropy-based test-time adaptation are strong, simple options, though the file has no reported results for them on these benchmarks. They need batches of target images or online gradient updates. Our rule works on a single image with frozen weights. We will include batch-norm recalibration as an extra reference that has access to unlabeled target batches. We treat it as an upper-bound-style comparison and not as an equal-information baseline. Matching it while using only single images would be a meaningful result.

Why not just scale up synthetic data? Driving in the Matrix shows that 200k synthetic images can beat Cityscapes training on KITTI vehicle detection (0.6803 vs 0.6247 mAP, Easy). The same paper acknowledges a residual gap. PASTA's analysis suggests more renders from the same engine add no missing high-frequency variance. We test this on Sim10k-scale and larger GTA-based detection training. Specifically, we check whether the gating gain shrinks as synthetic data grows, which would indicate a diminishing benefit.

Proposed Experiments

For segmentation, we train on GTA5 and on SYNTHIA and evaluate on Cityscapes, BDD100K and Mapillary Vistas, with mIoU as the metric. The file has no reported results on Lacuna for PASTA or SAN/SAW on BDD100K or Mapillary, so all such numbers will be produced by our reruns. For detection, we train on Sim10k and evaluate vehicle mAP on Cityscapes and KITTI. For classification, we use VisDA-17 synthetic→real, comparing against the 61.1% ASG figure. For mechanism parity, we measure decisive-feature distance on the 2,126 matched KITTI and VirtualKITTI2 pairs from the fidelity-gap study. We also report inference latency as a function of $K$.

Three checks decide the project. First, the instability–harm correlation on real Cityscapes images must be clearly positive and stronger than on synthetic validation images. Second, gating must beat matched test-time augmentation at equal compute. Third, gating must add a consistent gain on top of PASTA training. Passing the first two but failing the third would still be a useful finding: per-image reliability helps models trained without frequency augmentation but is redundant with it. Failing the first means the probe does not measure what we hoped, and the method should not be pursued in this form.