Models trained on rendered images, such as street scenes from the video game GTA5, lose much of their accuracy on real photographs. The usual fixes are heavyweight. Adversarial image translators restyle synthetic images to look real but sometimes warp shapes. Multi-stage self-training pipelines cost days of GPU time. This proposal starts from a narrower, testable hypothesis: a large part of the visual gap is a mismatch in fine texture. Rendered surfaces lack the irregular, self-similar mid-frequency detail of real foliage, asphalt and facades.
Fractional-order derivatives give a precise way to measure that detail. These are derivatives of non-integer order, say 0.4 or 1.5, that sit between "intensity", "slope" and "curvature". In image restoration, fractional-order total variation (a penalty on the size of these fractional derivatives) is known to keep fine textures that ordinary gradient penalties wipe out. We propose to measure how the distributions of fractional-derivative responses differ between synthetic and real images across a continuum of orders. Where they differ, we train a small translator, with no adversarial discriminator, that injects texture until those distributions match, while label boundaries from the simulator hold geometry fixed.
Core Claim
The claim has two parts, and the second is only pursued if the first holds.
The first part is diagnostic. The synthetic and real distributions of fractional-derivative responses should diverge most at low fractional orders (roughly 0.3 to 0.7), and the divergence should sit in their shape (how heavy-tailed or sparse they are) rather than in their variance. This prediction comes from Image Motion Restoration Using Fractional-Order Gradient Prior, a classical blind-deblurring method. It argues that natural textures are fractal-like, and that their fractional-order gradient responses look Gaussian rather than sparse. A renderer built from tiled texture maps and clean shading should produce sparser, more kurtotic responses at those orders.
The second part is about the method. If the diagnostic holds, a non-adversarial translator trained only to match these per-order distributions should close more of the GTA5→Cityscapes gap than either of two alternatives at matched cost. One alternative matches only second-order (spectral) statistics; the other is an adversarial translator.
What is new is the use of fractional calculus as a statistical measurement of domain shift and a distribution-matching target. In prior fractional-calculus work it is only a smoothness prior on a single image. In prior sim-to-real work, translation constraints are pixel-wise or adversarial.
Mechanism
A fractional derivative of order $\alpha$ along the horizontal axis can be computed exactly in the Fourier domain. One multiplies the image spectrum by $(i\omega_x)^\alpha$, where $\omega_x$ is horizontal frequency. This avoids the truncation error of finite Grünwald–Letnikov masks noted across the restoration literature. It also makes plain that each order is a power-law reweighting of frequencies. For an image $u$ and order $\alpha$, let $P_\alpha(u)$ denote the empirical distribution of the responses $|D^\alpha_x u|$ and $|D^\alpha_y u|$, pooled over pixels and over a batch. The translator $T$ is a shallow residual convolutional network that adds a correction to each synthetic image $x_s$. It is trained with
$$\mathcal{L}(T)=\sum_{\alpha\in A} \mathrm{SW}\big(P_\alpha(T(x_s)),\,P_\alpha(x_t)\big)+\lambda\,\big\|B_s\odot\big(\nabla T(x_s)-\nabla x_s\big)\big\|_1 .$$
Here $A$ is a grid of orders in $[0.2, 1.8]$. $\mathrm{SW}$ is the sliced Wasserstein distance between one-dimensional distributions, which is differentiable and cheap. $x_t$ are unlabeled real images. $B_s$ is a band of pixels around the semantic boundaries of the simulator's ground-truth labels, and $\nabla$ is the ordinary first-order gradient. The weight $\lambda$ trades texture freedom against geometric fidelity.
The second term means object contours must keep their edge strength, while surfaces between contours are free to gain texture. This targets the failure that Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks (PixelDA) tries to control. PixelDA restyles synthetic images with a generative adversarial network and protects shape with a masked pixel-wise squared-error loss. That loss penalizes texture injection and edge displacement equally. Our loss separates the two, and it removes the discriminator entirely.
The result is texture transfer driven by measured statistics: inspectable, cheap, and attributable to specific orders.
Research Program
Diagnostic phase. We compute $P_\alpha$ for every order in $A$ on GTA5, SYNTHIA and Virtual KITTI, and on Cityscapes and KITTI. We record per-order sliced Wasserstein distance, variance ratio and kurtosis ratio. We do this per semantic class using ground-truth masks, because texture-heavy classes (vegetation, terrain, building) should diverge more than flat or thin ones (road, pole, sign). Virtual KITTI→KITTI acts as a control. These are geometric clones of the same streets, so remaining divergence is mostly appearance. A second control computes the same divergence between two disjoint halves of Cityscapes, giving a floor.
The hypothesis fails if the divergence is flat across orders, or is fully explained by variance (which is a power-spectrum quantity). In that case we report the negative result and do not build the translator.
Translation phase. We train $T$ on GTA5 with unlabeled Cityscapes training images. We then train a fixed segmentation network on the translated images with source labels only, and evaluate on the Cityscapes validation set.
Integration phase. We feed translated images into an existing self-training method to test whether the gains add to or overlap with what self-training already recovers.
Three roles for prior work shape this program. Real order total variation with applications to the loss functions in learning schemes proves that real-order total variation is a well-posed loss: it is lower semi-continuous, including in the order itself, and has compactness guarantees. It runs no experiments. That supports sweeping and even learning the order set $A$, and this work supplies the missing empirical test, though in a distribution-matching role rather than as the plain regularizer the theory considers.
Learning Semantic Segmentation From Synthetic Data: A Geometrically Guided Input-Output Adaptation Approach. (GIO-Ada) conditions translation on simulator depth and aligns joint depth-and-semantics outputs adversarially. It shows that free simulator side information stabilizes translation. We reuse that lesson with semantic boundaries in place of depth, and drop the adversarial machinery.
Exploring High-quality Target Domain Information for Unsupervised Domain Adaptive Semantic Segmentation (EHTDI) pastes confident real-image pixels onto source images and adds contrastive losses. It documents that adapted models fail mainly at object boundaries. This constrains our design: the translator must not move boundaries, hence the boundary-band term.
Baselines
The naive reference point is source-only training: the same segmentation network, same ImageNet initialization, same schedule, trained on raw GTA5. VisDA: The Visual Domain Adaptation Challenge reports a source-only GTA5→Cityscapes mIoU (mean intersection-over-union) of 21.4 for its baseline. We will re-run our own source-only number, since the architecture differs. VisDA also warns that ImageNet pretraining can mask what adaptation contributes. We therefore report a randomly initialized backbone variant for the diagnostic correlations as well.
The strongest translation-level competitor is adversarial pixel translation in the PixelDA style, plus a CycleGAN-style translator with the same segmentation network trained on its outputs. PixelDA reports 98.2% on MNIST→MNIST-M and a 23.5° mean angle error on synthetic→real LineMod. No reported result was found on Lacuna for PixelDA on GTA5→Cityscapes, so we run it ourselves. For fairness, every translator gets the same unlabeled Cityscapes images, the same GPU-hour budget and the same number of hyperparameter trials. Beating it means higher source-only mIoU on Cityscapes validation across three seeds, with no lower boundary-band accuracy.
The strongest full-system competitors are EHTDI, which reports 58.8% mIoU on GTA5→Cityscapes with 63 hours on a single RTX 3090Ti, and Unsupervised Contrastive Domain Adaptation for Semantic Segmentation, which reports 60.2% mIoU on GTA5→Cityscapes. We do not expect translation alone to match these. The test is whether feeding our translated images into a reproduced EHTDI raises its reproduced score by more than seed-to-seed variation, at under 10% extra compute.
The ablations isolate the key ingredient, which is distribution shape at fractional orders. The first matches only per-order variance, which is equivalent to matching banded power spectra. The second restricts $A$ to integer orders $\{1, 2\}$. The third replaces fractional derivatives with a Laplacian pyramid filter bank of equal size. The fourth removes the boundary-band term.
Why not just…?
Why not just match the Fourier amplitude spectrum, or color histograms? Spectrum matching is cheap and close in spirit, since fractional derivatives are power-law frequency filters. But a spectrum captures only variance per frequency. The hypothesis drawn from Fu et al. is about tail shape, Gaussian versus sparse, which a spectrum cannot see. The variance-only ablation tests exactly this. If it matches the full method, then shape does not matter and the honest conclusion is that spectral matching suffices.
Why not just use any multiscale filter bank instead of fractional derivatives? This is the real threat. Fractional derivatives may be just one reasonable filter family. The Laplacian-pyramid ablation is designed to settle it. If the pyramid ties, the contribution shrinks: the diagnostic finding that the gap lives in higher-order texture statistics would remain, and the fractional family would be valued only for its single continuous parameter. We will say so.
Why not just use adversarial translation, which already learns arbitrary statistics? It can in principle. In practice, the file's account of PixelDA and GIO-Ada shows these methods need extra content or geometry constraints to stop shape distortion, and they offer no account of what was changed. The head-to-head at matched budget, together with boundary-band accuracy, decides whether explicit statistics lose any accuracy for their transparency and stability.
Why not just put a fractional total variation penalty on the target predictions to sharpen boundaries? It is the most obvious combination of the two directions. But a smoothness prior on softmax maps mostly duplicates what integer total variation or post-hoc conditional random fields already do. It also does not touch the appearance gap that causes the boundary errors. We leave it as a secondary comparison, not the core.
Decisive Checks
The project is decided by three results.
The first is whether the per-order divergence peaks at fractional orders and in shape rather than variance. It should be larger for texture classes and small for the Cityscapes split-half floor.
The second is whether, across the three domain pairs and multiple segmentation backbones, per-order divergence after translation predicts the source-only mIoU recovered. A clear rank correlation would make the measure useful as a cheap gap diagnostic even if the translator gains are modest.
The third is whether the full objective beats both the variance-only and filter-bank ablations by more than seed variance. It must do so on Cityscapes validation with per-class IoU reported, and replicate on SYNTHIA→Cityscapes, where GIO-Ada reports 37.3% on 16 classes. If the first check fails, the proposal stops there with a documented negative result about where the synthetic texture gap does not live.
References
- Image Motion Restoration Using Fractional-Order Gradient Prior
- Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks
- Real order total variation with applications to the loss functions in learning schemes
- Learning Semantic Segmentation From Synthetic Data: A Geometrically Guided Input-Output Adaptation Approach.
- Exploring High-quality Target Domain Information for Unsupervised Domain Adaptive Semantic Segmentation
- VisDA: The Visual Domain Adaptation Challenge
- Unsupervised Contrastive Domain Adaptation for Semantic Segmentation