Intrinsic decomposition means splitting a photograph into the color of the surfaces (albedo) and the light falling on them (shading). The task is ambiguous at the level of a single pixel: a dark patch may be a dark material in light or a light material in shadow. Two separate walls of the same room, or four legs of one chair, should share an albedo even when a shadow or an occluding object sits between them. A convolutional network sees only a small neighborhood at each layer, so it cannot easily enforce that sharing. A transformer can link distant pixels, but when it is trained on synthetic renders it is free to link them for the wrong reasons, such as renderer-specific texture and noise statistics that do not exist in real photos.
We propose a network whose long-range pathway can only pass albedo information between pixels that a real-image-trained semantic segmenter believes belong to the same kind of surface. The local convolutional pathway keeps sharp material edges. The long-range pathway reads from a small set of per-class summary features, and routing weights come from the segmenter's class probabilities, so the set of pixels that exchange information is decided by a signal trained on real labels rather than by the synthetic data. The central bet is about where the sim-to-real gap lives. We expect it to live mostly in local appearance statistics, while the grouping "these regions are the same kind of stuff" transfers across domains. If that is right, long-range routing that is restricted by grouping should transfer from synthetic training to real images better than either a pure convolutional network or unrestricted attention of the same size.
Core Claim
Trained only on synthetic data, a hybrid convolution-plus-global network helps on real images only when its global pathway is constrained to route along semantic material groups. Unconstrained global attention of equal capacity will fit synthetic shortcuts and gain little or nothing on real benchmarks. The improvement from gated routing should be concentrated on pixel pairs that are far apart in the image. It should not come at the cost of over-smoothing surfaces that share a class label but differ in color.
The claim is falsifiable in three ways, each measured on Intrinsic Images in the Wild (IIW, a real-photo benchmark scored by the Weighted Human Disagreement Rate, WHDR, the fraction of human pairwise "which point has darker surface color" judgments the model gets wrong):
- No transfer advantage: gated routing does not beat unrestricted attention.
- Wrong locus: the gains are uniform across pair distance rather than concentrated on distant pairs.
- Over-smoothing: the gains disappear once surfaces that share a class but differ in albedo are isolated.
Relation to Prior Work
Several papers motivate the premise that semantic grouping constrains reflectance.
- Higher Order Priors for Joint Intrinsic Image, Objects, and Attributes Estimation penalizes reflectance variance inside object or material regions.
- Semantic Priors for Intrinsic Image Decomposition uses object proposals and deep-feature similarity to tie together disconnected parts of one object, reaching a WHDR of 17.72 on IIW. It does this inside a separate, non-learned iterative optimizer.
- Joint Learning of Intrinsic Images and Semantic Segmentation shares a convolutional encoder across albedo, shading, and segmentation decoders and reports a threefold albedo error reduction on its synthetic garden dataset. It was never evaluated on real images, and its purely local operations cannot connect distant regions.
- The multi-view inverse rendering systems Factorized Inverse Path Tracing for Efficient and Accurate Material-Lighting Estimation and Multi-view Inverse Rendering for Large-scale Real-world Indoor Scenes apply semantic-mask consistency to real rooms. They require accurate 3D meshes and multi-exposure capture.
Our proposal moves the grouping prior inside a single-image network as a soft, learned routing constraint. Neither a hard penalty nor a 3D pipeline is involved.
Other papers supply the architectural pieces.
- CCNet: Criss-Cross Attention for Semantic Segmentation shows that cheap global context can be bolted onto a convolutional backbone (81.9% mIoU on Cityscapes). It also gives us a segmenter trained on real ADE20K scenes, which we reuse as the grouping signal.
- Bracketing Image Restoration and Enhancement with High-Low Frequency Decomposition assigns high frequencies to convolutions and low frequencies to attention.
- ClassLIE: Structure- and Illumination-Adaptive Classification for Low-Light Image Enhancement pairs a transformer branch for illumination with a convolutional branch for texture.
Neither of these two hybrids estimates physical material properties or restricts its attention by semantics.
Two papers constrain the design.
- A Guide to Robust Generalization: The Impact of Architecture, Pre-training, and Optimization Strategy finds that large ConvNeXt convolutional networks are more robust under perturbation than pure vision transformers. Hybrids win mainly at small scale (5–10M parameters). This is a direct warning that adding attention can hurt out-of-distribution behavior, so our main comparison fixes parameter count and includes a strong pure-convolution control.
- Intrinsic Image Decomposition Using Paradigms argues the opposite of our semantic premise. It trains on abstract Mondrian albedo and Perlin-noise shading, reaches 16.86% WHDR on IIW, and argues that photorealistic synthetic data hurts real performance. It also concedes large-scale spatial drift that it patches with expensive tile averaging at inference.
We take that paper seriously: its local-statistics position is one of our baselines, and its training data is one of our training variants.
Research Program
The network is a convolutional encoder-decoder with skip connections, which carries edges and fine texture. A global module sits at the bottleneck and at one intermediate scale. A frozen segmenter trained on real ADE20K produces a class probability vector $p_i$ over $K$ classes for each pixel $i$. From encoder features $f_i$ we form one summary feature per class, $c_k = \sum_i p_{ik} f_i / \sum_i p_{ik}$, a soft average of the features of pixels believed to be class $k$. Each pixel then reads back a mixture of these summaries, weighted by its own class probabilities and a learned compatibility score:
$$ g_i = \sum_{k=1}^{K} \frac{p_{ik}\, \exp(q_i^\top W c_k)}{\sum_{k'} p_{ik'}\, \exp(q_i^\top W c_{k'})}\, c_k . $$
Here $q_i$ is a query projection of $f_i$, and $W$ is a learned matrix. The factor $p_{ik}$ restricts where information can come from. The compatibility term lets the network decline to share within a class, for example between two walls painted different colors. Cost is linear in pixel count times $K$, cheaper than criss-cross attention. The output $g_i$ is added to the decoder input, and the decoder predicts log-albedo, with shading obtained as image divided by albedo.
We train on Hypersim, which provides diffuse reflectance, diffuse illumination, and semantics for 77,400 photorealistic indoor renders (Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding). The same frozen segmenter is applied to both synthetic training images and real test images. The grouping signal therefore comes from one fixed real-trained model rather than from ground-truth synthetic labels that are unavailable at test time. A second training variant mixes Paradigms-style abstract images into the synthetic set. This tests whether abstract local statistics and semantic routing are complementary, which is the unresolved tension between the Paradigms work and the semantic-prior work. No real intrinsic labels are used in training.
The analysis component is as important as the model. We bin IIW pairwise judgments by image distance between the two points. We also split them by whether the two points fall in the same predicted segmenter class, crossed with whether humans judged them equal or different in albedo. This yields a specific diagnostic set: pairs in the same class that humans nevertheless judged different. That set is exactly where semantic regularization should fail if it over-smooths.
Baselines
Naive reference. The same convolutional encoder-decoder with the global module removed, trained on Hypersim with the identical loss, schedule, and augmentation. Every other variant is measured as a difference from this network.
Strongest external number. The strongest reported IIW number in our file is 15.2% WHDR from GLoSH: Global-Local Spherical Harmonics for Intrinsic Image Decomposition. Paradigms reports 16.86%, and Semantic Priors reports 17.72. ReasonX: MLLM-Guided Intrinsic Image Decomposition reports only a relative 9–25% WHDR reduction over its base models, with no absolute value in our file. Deep Hybrid Real and Synthetic Training for Intrinsic Decomposition claims best-in-class WHDR, but no reported result for it was found on Lacuna. These systems use different training data, some of it real. We therefore treat 15.2% as a reference point: matching it with synthetic-only training would be a strong result, but it is not the decisive test.
Strongest controlled competitor. A parameter-matched ConvNeXt-style convolutional network with large kernels and dilation, motivated by the robustness findings of A Guide to Robust Generalization. Since the proposal's premise is that local operators cannot do long-range sharing, this is the control most likely to embarrass it.
Unconstrained attention of equal capacity. A version with CCNet criss-cross attention, and a version with the same per-class-summary module but a learned, ungated clustering in place of $p_{ik}$. These are the key ablations: they keep global routing and remove only its dependence on real-trained semantics.
Explicit semantic regularizer. The naive network trained with a within-segment albedo variance penalty, and separately with post-hoc averaging of predicted albedo inside segments. This is the classical higher-order-prior idea implemented cheaply.
Fairness and the success criterion. All controlled variants share the training data, input resolution, frozen segmenter access where applicable, a parameter count within 5%, and an equal hyperparameter search budget over learning rate and regularizer weight. Tuning uses a Hypersim validation split only, never IIW. Beating the competitors means lower IIW WHDR than every controlled competitor across three seeds, with the margin exceeding seed spread. Concretely, the proposal counts as successful only if it improves over both the ConvNeXt control and the ungated ablation. It must also not lose to the explicit regularizer on the same-class-different-albedo pairs.
Why not just…?
Why not just use a bigger convolutional network with dilation or large kernels? This is the most serious alternative. Robustness evidence favors large ConvNeXt models over transformers (A Guide to Robust Generalization), and receptive field alone may suffice for rooms at IIW resolution. We do not argue it away; the parameter-matched ConvNeXt control settles it. If that control matches us on distant IIW pairs, then long-range sharing is not the bottleneck and the claim fails.
Why not just smooth albedo within segmentation masks, as a loss or post-process? This is exactly the higher-order prior of Higher Order Priors for Joint Intrinsic Image, Objects, and Attributes Estimation, and it is cheap. Its weakness is that it cannot tell a uniform wall from a patterned rug of the same class, so it should lose accuracy on same-class-different-albedo pairs. Our diagnostic split measures precisely this. If the post-process matches us overall and on that split, the learned compatibility term is unnecessary and we would report the simpler method.
Why not just add standard attention, since hybrids already work in image restoration? The hybrids in ClassLIE and HLNet are trained and tested within one domain. Our question is transfer from synthetic data, where unrestricted attention can learn renderer shortcuts. The criss-cross and ungated-summary ablations test this directly. Our prediction is that they fit Hypersim validation as well or better but transfer worse to IIW. If they transfer equally well, the gating hypothesis is wrong.
Why not just train on abstract Paradigms data and skip semantics? Intrinsic Image Decomposition Using Paradigms gets 16.86% WHDR this way, but it admits large-scale drift fixed by costly tile averaging. The mixed-training variant tests whether our routing removes that drift without tile averaging, which would show up as gains on distant pairs.
Why not just translate synthetic images toward real style? Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks itself limits the approach to low-level appearance changes. Altering pixels also breaks the link between the image and the albedo-times-shading ground truth. We therefore treat it as outside the main comparison.
Decisive Checks
We will report WHDR on the IIW test split for all variants, plus the shading precision measure of the Shading Annotations in the Wild benchmark. We also report Hypersim held-out albedo error, so that the synthetic-to-real gap is visible for each architecture as the difference between in-domain and real performance.
Three results decide the proposal:
- Transfer. Gated routing beats both the ConvNeXt control and the ungated ablations on IIW.
- Locus. Its advantage over the naive network grows with pair distance, while short-distance pairs are roughly unchanged. This is what distinguishes long-range material sharing from generic capacity.
- No over-smoothing. On same-class pairs that humans judged different, gated routing is at least as accurate as the naive network and more accurate than mask averaging.
A further check swaps the real-trained ADE20K segmenter for one trained on Hypersim labels. If the grouping signal's real-image provenance matters, as we hypothesize, performance should drop. If it does not drop, semantic grouping transfers regardless of where the segmenter was trained, and the design simplifies accordingly.
References
- Higher Order Priors for Joint Intrinsic Image, Objects, and Attributes Estimation
- Semantic Priors for Intrinsic Image Decomposition
- Joint Learning of Intrinsic Images and Semantic Segmentation
- Factorized Inverse Path Tracing for Efficient and Accurate Material-Lighting Estimation
- Multi-view Inverse Rendering for Large-scale Real-world Indoor Scenes
- CCNet: Criss-Cross Attention for Semantic Segmentation
- Bracketing Image Restoration and Enhancement with High-Low Frequency Decomposition
- ClassLIE: Structure- and Illumination-Adaptive Classification for Low-Light Image Enhancement
- A Guide to Robust Generalization: The Impact of Architecture, Pre-training, and Optimization Strategy
- Intrinsic Image Decomposition Using Paradigms
- Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding
- GLoSH: Global-Local Spherical Harmonics for Intrinsic Image Decomposition
- ReasonX: MLLM-Guided Intrinsic Image Decomposition
- Deep Hybrid Real and Synthetic Training for Intrinsic Decomposition
- Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks