Routing Synthetic-to-Real Transfer Through Geometry in a Unified Vision Model

Rendered training data comes with perfect labels for depth, surface orientation and object masks. Models trained on it still break on real photographs, mostly because rendered pixels differ from camera pixels in texture, noise and lighting. This proposal starts from a simple observation. The labels themselves differ between the two worlds much less than the images do. A depth map of a rendered living room looks much like a depth map of a real one; the RGB images do not.

Recent "generalist" vision models make that difference usable. These are single networks that turn every modality (RGB, depth, surface normals, segmentation) into tokens in one shared space and can predict any subset from any other subset. Such a model can do something a stack of task-specific networks cannot do natively. It can predict the geometry of a real image, then predict semantics from that geometry, mostly bypassing the domain-sensitive RGB path. We propose to measure how the synthetic-to-real gap is distributed across modalities inside such a model. If it is concentrated in RGB, as we expect, we then test whether routing inference and cheap real-world supervision through the geometric modalities closes the gap better than standard domain adaptation.

Core Claim

In an any-to-any masked multimodal model trained on photorealistic synthetic indoor data, the domain gap lies mainly in the RGB input path. Geometric modalities, and the mappings between them and semantics, transfer almost unchanged. This has two consequences. First, chained inference (RGB to predicted depth and normals, then semantics conditioned on those predictions with the RGB tokens partly withheld) should beat direct RGB-to-semantics prediction on real images with no real labels at all. Second, adding real data that has only RGB plus raw sensor depth, which is cheap to collect because no human annotates it, should transfer semantic accuracy to the real domain through the shared token space. This is what makes the proposal a claim about architecture and not just about data. Task-specific heads cannot condition one output on another output unless someone builds a separate pipeline for that pairing. In a unified model, that conditioning comes for free.

Relation to Prior Work

Two papers motivate the premise. Ren & Lee's cross-domain multi-task learning trains an AlexNet with separate heads on synthetic depth, normals and contours. It aligns features to real Places365 images with an adversarial discriminator, and it finds that multi-task geometric supervision avoids shortcuts that single-task training falls into. We keep their intuition that geometry transfers, but we exploit it at inference time through a shared token interface rather than only as a pretraining signal. Hypersim supplies photorealistic synthetic scenes with dense depth, normals and instance labels, and shows that pretraining on them helps NYUv2. Its evaluation uses task-specific fine-tuning and never asks how the gap differs between modalities.

Three papers supply the architecture and a constraint on it. 4M trains one transformer to reconstruct randomly masked subsets of tokenized RGB, depth, normals, semantics and text; that masked any-to-any training is the mechanism we reuse. UNIFIED-IO and LVM cast tasks as discrete token sequences. LVM explicitly reports that its image tokenizer "sometimes" struggles with synthetic data, which warns that the tokenizer may itself be where the gap lives. We therefore measure the gap at the tokenizer and at the transformer separately.

Two further papers are the nearest adjacent work, and they constrain the design. S4T shows that when multi-head models adapt at test time across domains (Taskonomy to NYUD-v2), tasks adapt at different speeds: improving one degrades another. It fixes this with a masked module that ties task predictions together. We share its view that cross-task relations are the stable thing to anchor on. S4T works on a head-based ResNet and adapts weights at test time; we instead change what the model conditions on, and we use a synthetic source. Driving with DINO finds that low-level conditioning signals carry synthetic texture into the output ("texture baking"), while high-level signals like depth and segmentation transfer but lose detail. It resolves this for video generation by keeping only the major principal components of DINOv3 features. Its dilemma is a direct warning for us: chaining through depth may lose the fine detail that semantics needs. That is why our chained inference only partly withholds RGB rather than dropping it.

Research Program

We first build a 4M-style model at moderate scale. It covers RGB, depth, surface normals and 40-class semantic segmentation, each with its own tokenizer, trained on Hypersim with 4M's input-and-target masking. We train two variants. One is purely synthetic, tokenizers included. The other initializes the RGB path from a generic image encoder, so we can tell whether the effect survives a real-image prior.

The first analysis measures the gap per modality at two levels. At the tokenizer level, we compare reconstruction error and code-usage histograms on Hypersim versus NYUv2 for each modality. At the model level, we compute a per-modality shift score:

$$\Delta_m = \mathrm{FD}\big(h_m(\text{Hypersim}),\, h_m(\text{NYUv2})\big),$$

where $h_m$ is the transformer encoder's pooled embedding of modality $m$'s tokens, and FD is the Fréchet distance between the two embedding distributions. FD is the same Gaussian-fit distance behind FID. We compute it on scene-matched subsets so that differences in scene category do not dominate.

NYUv2 depth comes from a real sensor and has holes and noise, so real depth is not perfectly clean. We report $\Delta_{\text{depth}}$ on raw sensor depth and on hole-filled depth, and we treat sensor artifacts as their own small gap. The premise holds if $\Delta_{\text{RGB}}$ is several times larger than the geometric and semantic scores. If it is not, the chaining story fails and we report that.

The second part is chained inference. For a real image $x$, the model predicts depth and normal tokens $\hat g$. It then predicts semantics conditioned on $\hat g$ together with a random fraction $\rho$ of the RGB tokens. We sweep $\rho$ from 0 (pure geometry) to 1 (direct prediction). We expect the best $\rho$ to sit in the interior, trading the RGB gap against the detail loss that Driving with DINO warns about. Choosing $\rho$ uses only synthetic validation data plus an unsupervised agreement signal on real images: consistency between the direct and chained predictions. We never use real labels for this choice.

The third part is partial-modality real training. We continue training on a mix of Hypersim (all modalities) and NYUv2 raw frames that carry only RGB and sensor depth, with no semantic labels. Masked modeling then learns the real mapping between RGB and depth directly, while the mapping from depth to semantics stays anchored in synthetic data.

Baselines

The naive reference is the same unified model trained on Hypersim and evaluated directly on NYUv2 with RGB-only input and no adaptation. This number sets the size of the gap. It also doubles as the ablation that isolates our key ingredient, since it differs from our method only in whether predicted geometry sits in the conditioning set. We run it at $\rho=1$ with identical weights.

The strongest domain-adaptation competitor is DAFormer. It is a transformer segmenter trained by self-training on unlabeled target images, with rare-class sampling and a feature-distance regularizer toward ImageNet features. The only numbers the file reports for it come from driving: 68.3 mIoU on GTA5 to Cityscapes. We found no reported result on Lacuna for DAFormer on Hypersim to NYUv2. We will run it from the official codebase on Hypersim to NYUv2 semantics. It gets the same unlabeled NYUv2 RGB frames and a matched number of GPU-hours, and we tune its hyperparameters with the same search budget we give our model's $\rho$ and mixing ratio.

The strongest multi-task competitor is S4T. It is a multi-head ResNet with a test-time masked synchronizer, and it reports a 34.94% relative improvement ($\Delta_{\text{TTT}}$) on Taskonomy to NYUD-v2, against 28.47% for TTTFlow. We run it from its code with Hypersim as the source. We also run our method on Taskonomy to NYUD-v2 so that it can be compared against S4T's own reported setting.

The ablation that separates unification from mere chaining is a two-stage specialist pipeline. One network maps RGB to depth and normals. A second maps RGB plus geometry to semantics, with RGB dropout matched to $\rho$. Both are trained on the same Hypersim data and given the same parameter budget. For partial-modality training, the matching control is a multi-head model with real sensor depth added as an auxiliary loss.

As upper references trained on real data, we cite 4M-L's 94.4% δ1 depth accuracy and Painter's 0.288 RMSE on NYUv2, both from the file. Neither comes from a comparable synthetic-only protocol.

We count our method as beating a baseline if it exceeds that baseline on NYUv2 semantic mIoU. For depth and normals, it must not fall behind: we require no degradation beyond the variation across three random seeds, which rules out the S4T-style trade-off.

Why not just…?

Why not just run DAFormer-style self-training per task? This is the main practical threat, and our experiments test it directly rather than assuming it away. The file gives no evidence of how self-training behaves on indoor synthetic data with 40 classes. Self-training also yields one model per task, which is exactly the maintenance burden the unified line tries to remove. If DAFormer matches our semantic mIoU, that is still informative: the geometry route adds nothing beyond pseudo-labeling. We would then test whether the two methods stack.

Why not just use frozen foundation features with the PCA pruning of Driving with DINO? That filter was validated only as conditioning for video generation. It deliberately discards fine detail, which discriminative segmentation needs. We include a variant that trains segmentation and depth heads on pruned RADIO or DINO features from Hypersim. If it wins, the conclusion is that the gap is best fixed in feature space rather than by routing through geometry, and we would say so.

Why not just a multi-head network with real depth as an auxiliary loss, or a two-stage chain? These controls are exactly the specialist pipeline and multi-head auxiliary baselines described above. They separate the two things our model bundles: chaining through geometry, and using a shared token space. If the two-stage specialist matches us, the benefit comes from chaining, and unification is only a convenience. We consider this a plausible outcome and a publishable one.

Why not just render more photorealistically? Hypersim is already ray-traced and physically factorized, and the gap on it remains. We report the gap on Hypersim itself rather than arguing about future renderers.

Decisive Checks

The benchmark is Hypersim as source and NYUv2 as target, for semantics (mIoU), depth (δ1 and RMSE) and normals (mean angular error), with Taskonomy to NYUD-v2 as a secondary benchmark that allows direct comparison with S4T. Three results would settle the proposal.

First, the per-modality gap. If $\Delta_{\text{RGB}}$ does not clearly dominate, the premise is wrong.

Second, the $\rho$ sweep. It should show an interior optimum that beats $\rho=1$ on real semantics without any real labels. If $\rho=1$ is best, geometry routing does not help.

Third, partial-modality training. Real RGB plus sensor depth should raise semantic mIoU more than the same data does in the multi-head auxiliary model and the two-stage specialist control. That result is the one that would justify the unified architecture specifically.

We also plan one diagnostic tied to the texture-baking finding. We re-run the chained model with synthetic RGB tokens swapped in from a mismatched scene. If semantic predictions still follow the geometry, the model has learned to rely on the domain-robust path.