Telling Shadow from Black Rubber: Language-Informed Albedo Priors for Near-Infrared Shape-from-Shading

Near-infrared (NIR) shape-from-shading recovers fine 3D surface detail from a single image by exploiting a physical accident. Shape-from-shading means inferring surface orientation from how bright each pixel is. At wavelengths of roughly 700 to 1100 nm, most dyes, pigments and skin chromophores reflect almost uniformly. Pair a narrow-band filter with an active NIR flash and ambient light is also rejected. The result is that brightness depends almost only on how each surface patch faces the flash, whatever the room lighting. The trick has a known hole. Materials pigmented with carbon black, such as tires, black rubber, carbon fiber and many black plastics, absorb NIR as strongly as visible light (Material-Based Object Segmentation Using Near-Infrared Information). A dark NIR pixel can therefore mean two things: the surface is turned away from the light, or it is made of carbon black. Every shape-from-shading method that assumes flat NIR albedo will bend such surfaces into false geometry.

We propose to settle that ambiguity with what the scene is, not how it is shaded. A camera that captures RGB and NIR through a shared lens gives, at each pixel, a visible-light view that open-vocabulary recognition can name ("black rubber tire," "cotton shirt," "matte plastic"). Text descriptions of those materials carry prior knowledge about NIR reflectance that a closed label set cannot extend to new materials. We use these language-derived labels to predict where the flat-albedo assumption fails and by how much. The shading solver then treats dark pixels in those regions as low albedo rather than steep geometry. The claim is narrow and testable: a per-region albedo prior derived from language embeddings reduces normal error on NIR-absorbing materials. It keeps the lighting invariance of the NIR channel, and it generalizes to absorbing materials never named during training, which a one-hot material classifier cannot do.

Core Claim

Write the flash-lit NIR image as $I(x) = \rho(x)\,\max(0, n(x)\cdot l(x))/d(x)^2 + \epsilon$. Here $\rho(x)$ is NIR albedo, $n(x)$ the surface normal, $l(x)$ and $d(x)$ the calibrated flash direction and distance, and $\epsilon$ sensor noise. Existing learned NIR shape-from-shading either treats $\rho$ as constant or absorbs it into a network without any material knowledge. Fine-Scale Surface Normal Estimation Using a Single NIR Image trains an adversarial network on 64×64 NIR patches and is fully category-agnostic. A Dark Flash Normal Camera conditions a two-branch U-Net on a closed-set face segmentation mask, for a single material family. Neither can handle an absorber it has not been told about.

We replace the constant with a prior $\hat\rho(x) = g(e(x))$. The term $e(x)$ is the text embedding of the material description matched to the region containing $x$, and $g$ is a small learned map from embedding to expected NIR albedo and prior confidence. The prior enters the photometric loss with a confidence weight, so a wrong or uncertain label falls back to the learned, category-agnostic albedo instead of forcing an error. The prediction is specific. Gains should appear only on NIR-dark materials, should persist for materials held out of training, and should shrink as the RGB image degrades, because the semantics come from RGB even though the shading does not.

Research Program

The system has three parts, and the scale question sits in how they are split.

The first part is the region-labeling pass. It runs a frozen vision-language encoder on the RGB image and assigns each image patch a material description by patch-to-text similarity. We label at the patch level, not with the global summary token. [CLS] is Not Enough shows that the global token of CLIP-style encoders sees mainly the dominant object: patch-level inference with adaptive aggregation raises MS COCO multi-label mAP from 49.2% to 68.8%. Black gaskets, handles and cables are exactly the small, non-dominant objects the global token misses. We reuse that paper's category-wise patch aggregation directly.

The second part addresses a limitation the same work flags: 14×14 or 16×16 patch grids are too coarse for material boundaries. We therefore upsample patch labels to pixel resolution using NIR edge guidance. On a coaxial rig the two images are pixel-registered, so this step needs no alignment correction.

The third part is the shading network. It works on high-resolution local NIR crops, as in the Yoon et al. design, and also receives a low-resolution global pass. The global pass constrains the low-frequency shape drift that paper reports, and it is where the per-region albedo prior is pooled, so that a single mislabeled patch cannot flip a whole surface.

Illumination robustness is handled as a constraint, not a module. The NIR shading channel is lighting-invariant by construction: A Dark Flash Normal Camera reports roughly 12.08° to 12.10° mean angular error across well-lit, shadowed, overexposed and low-light faces. The semantic channel inherits the fragility of RGB. We measure how quickly the albedo prior degrades as RGB light drops. We also test whether zero-shot labels computed on the NIR image itself serve as a usable fallback, in the spirit of the RGB-thermal open-vocabulary work in OpenRSS and Open-RGBT. Those systems inject a second spectral band into CLIP/SAM pipelines for segmentation, but they never model shading or geometry.

Data is the main practical cost. No dataset in the file combines pixel-aligned RGB and NIR, dense ground-truth normals, and carbon-black materials. We will render a synthetic set with measured NIR reflectance spectra, including carbon-black plastics and rubbers alongside NIR-bright dyed fabrics, captured under a simulated flash with varied ambient RGB lighting. We will then validate on real data at two scales. The first is the Pixel-aligned RGB-NIR Stereo Imaging and Dataset for Robot Vision: 90,000 real coaxial frames with LiDAR and 2,238 synthetic scenes with dense depth, from which coarse normals can be derived. Tires, cables and black equipment are common in its underground-parking scenes. The second is a small new capture of held-out absorbing objects with structured-light ground truth, using the same coaxial design.

Baselines

The naive reference is the flat-albedo assumption itself: the same shading network trained without any albedo prior. On the 101-object NIR benchmark, its closest published instance is Yoon et al., which reports 15.56° mean angular error over all views and 4.75° on center views (Fine-Scale Surface Normal Estimation Using a Single NIR Image). We will reimplement it on our data rather than compare against those numbers across datasets.

The strongest existing method is the dark-flash design of A Dark Flash Normal Camera. It takes RGB, NIR and a segmentation map as inputs and learns a separate reflectance branch. It is the most direct competitor because it already fuses RGB with flash NIR and already learns spatially varying albedo. It reports 12.08° in good light and about 12.10° under extreme light on its face dataset, against nearly 20° for an RGB-only model. We will retrain its architecture on our multi-material data with the same inputs, compute and tuning budget as our model, replacing face segmentation with the same material segmentation our system uses but as one-hot masks.

The key ablation is exactly that one-hot variant set against our text-embedding variant, with everything else identical. It isolates whether language adds anything beyond "a segmentation label was available." A second ablation removes the global low-resolution pass, to separate low-frequency drift correction from the albedo prior. For RGB-only single-image normal models of the foundation-model kind, we found no reported result on Lacuna for our setting. We will run the best available checkpoint zero-shot and fine-tuned on the same training split.

Beating the baselines means the following. On NIR-dark regions, error must drop with respect to the flat-albedo model and the one-hot dark-flash variant, with confidence intervals over scenes that do not overlap. On NIR-bright regions there must be no loss. On held-out absorbing materials, the text variant must beat the one-hot variant, which cannot represent a label it never saw.

Why not just…?

Why not just feed RGB and NIR into one network and let it learn albedo? This is the most serious threat. With pixel-aligned inputs, a network can learn that black, rubbery-looking regions are NIR-dark without any language at all. The dark-flash design is essentially this network. We expect it to match our method on materials seen in training. The claim therefore rests on the held-out-material split. If the plain fused network generalizes to unseen absorbers as well as the text-conditioned one, the language component is unjustified, and we will report that and fall back to the simpler design.

Why not just use a large RGB-only normal estimator? RGB-only prediction degrades under the lighting shifts where NIR helps. A Dark Flash Normal Camera measures a rise to nearly 20° under low light and overexposure versus about 12.1° with the NIR flash. Large web-trained models may close part of that gap on coarse shape. We will run one across our RGB lighting sweep. We expect it to remain competitive in good light on low-frequency shape and to lose on fine detail and in dark scenes; if it wins in the dark as well, the NIR premise is weaker than the literature suggests.

Why not just fuse coarse depth from stereo or LiDAR to fix the dark regions? Yoon et al. themselves call for combining rough global geometry with local shading detail. Coarse depth would indeed correct the gross warping of a tire's global shape. But it cannot tell whether fine dark structure (tread, seams, cable braiding) is geometry or albedo. Consumer depth also struggles on dark and specular surfaces (Or-El et al.). We will include a depth-fused variant using the rig's stereo depth and report fine-scale and coarse error separately. Our claim targets the fine-scale term.

Proposed Experiments

The decisive test is the held-out-material split. We will train on a set of named materials and test on absorbing materials whose names and appearances never appear in training, such as carbon fiber and black silicone, while holding lighting fixed. Our model must beat the one-hot and fused-RGB-NIR baselines on mean angular error inside those regions. We also score a direct diagnostic: per-pixel classification of dark NIR pixels as "attached shadow" versus "low albedo," with ground truth from rendering. That metric targets the ambiguity itself, where global normal error could hide a small but real effect.

The second experiment sweeps RGB illumination from well-lit down to near darkness while holding the NIR flash constant. It plots region-level normal error for the flat-albedo, one-hot and text-conditioned models, and for the NIR-labeled fallback. We predict the text prior's advantage decays with RGB light. The crossover point tells a practitioner when the semantic channel stops paying for itself.

The third experiment separates error by frequency band, using low-pass and high-pass components of the normal field. It checks that the global pass fixes low-frequency drift without blurring the fine detail that motivates NIR shading.

Finally, on the real coaxial dataset we report normals checked against LiDAR-derived coarse geometry. On our small structured-light capture we report full-resolution normal error. The capture is small, so the real-data results will be reported with per-object breakdowns rather than a single averaged number.