Pixel-Level Task Routing for Active Stereo in Sunlight

Depth cameras like the Intel RealSense project an invisible infrared dot pattern onto the scene and match those dots between two cameras. Indoors this works well. In sunlight the dots drown, and the network trained to match them starts learning from noise. Meanwhile the same infrared images carry ordinary scene texture, lit by the sun, which a conventional stereo matcher could use. Most networks for these sensors are trained without ground-truth depth. They learn by checking whether the predicted depth correctly warps one image onto the other, a "photometric reconstruction" loss. That loss can be computed on the projected pattern, the active cue, or on the sunlit texture, the passive cue. Which cue gives a useful training signal changes from pixel to pixel within a single frame. A wall in shade is pattern-dominated. The sunlit floor next to it is texture-dominated.

We propose to treat active and passive reconstruction as two tasks in a multi-task network, meaning one shared feature extractor trained on several objectives at once. The key change is that their relative weight is set per pixel by a predicted pattern-to-ambient ratio, not by a single number per task. That ratio is learned by a third, auxiliary task: splitting each infrared image into its projected-pattern component and its ambient component. The supervision for this task comes for free from occasional frames captured with the projector switched off. The claim is that, in this sensing problem, negative transfer between tasks happens at the level of individual pixels and is governed by a measurable physical quantity. Standard multi-task balancing methods, which reweight whole tasks, therefore cannot remove it.

Core Claim

Existing active-stereo learners take one of two positions on the ambient image. ActiveStereoNet trains disparity plus an invalidation mask with a self-supervised active photometric loss. It first applies local contrast normalization to strip low-frequency ambient light, so it discards exactly the texture that survives in sunlight, and it was evaluated only indoors. Connecting the Dots keeps the ambient component, but only to decode edges that sharpen depth boundaries. It assumes the dots are always visible.

ACDC-Net comes closest to our question. It computes reprojection losses separately on active and passive frames and takes a per-pixel minimum, adds SLAM landmarks, and acknowledges that patterns weaken in bright light. Its experiments, however, stay in indoor offices and apartments. A min-operator selects whichever loss is lower, and under washout a noise-dominated pattern loss can be low for the wrong reason.

Generic multi-task optimizers such as RotoGrad equalize gradient magnitudes and directions per task. If the active loss helps in the shaded half of an image and hurts in the sunlit half, no per-task weight or rotation can serve both halves.

Our hypothesis is that gating the two reconstruction losses per pixel, using an explicitly predicted pattern-to-ambient ratio $\hat\rho(p)$, will do two things. It will reduce error in high-ambient regions without giving up the indoor precision that active matching provides. It will also beat both learned and min-operator weighting, because the gate is anchored to a physical measurement rather than to the loss it is weighting.

Research Program

The network takes both the raw infrared stereo pair and its contrast-normalized version, so neither cue is thrown away at the input. A shared encoder feeds three heads: a disparity head, a decomposition head, and a confidence head.

The decomposition head predicts, for each pixel of a single projector-on frame $I_{on}$, the pattern component $\hat P$ and ambient component $\hat A$. It is supervised on the sparse subset of frames where a matched projector-off frame $I_{off}$ exists:

$$\mathcal{L}_{dec} = \lVert \hat A - I_{off} \rVert_1 + \lVert \hat P - (I_{on} - I_{off}) \rVert_1 .$$

This is the feed-forward, single-frame counterpart of the ambient/direct split that Neural Fields for Structured Lighting solves per scene offline. That method needs static multi-view strobed captures and, by its authors' account, struggles in bright outdoor light. We borrow its decomposition and drop the per-scene optimization. Subtracting on/off frames at test time is not an option, because it halves the frame rate and breaks on moving scenes, the motion problem also noted for Structured Light in Sunlight.

From the head's output we form $\hat\rho(p) = \hat P(p)/(\hat A(p)+\epsilon)$ and train disparity with

$$\mathcal{L}_{depth} = \sum_p \big[\, w(p)\,\mathcal{L}_{act}(p) + (1-w(p))\,\mathcal{L}_{pas}(p) \,\big], \quad w(p) = \mathrm{sg}\!\left[\sigma\big(\alpha(\log\hat\rho(p) - \tau)\big)\right].$$

The terms are defined as follows. $\mathcal{L}_{act}$ is the contrast-normalized pattern reconstruction loss of ActiveStereoNet. $\mathcal{L}_{pas}$ is a reconstruction loss on the ambient-dominated image with the pattern filtered out, as in ACDC-Net's smoothness term. $\sigma$ is the logistic function, $\alpha$ and $\tau$ are a slope and threshold, and $\mathrm{sg}$ is a stop-gradient. The stop-gradient matters: the disparity loss must not be able to push $\hat\rho$ toward whichever cue is easier to fit. The gate is the only multi-task mechanism added. The decomposition task shares the encoder because we expect pattern-versus-ambient features to be the features the disparity head needs, in the spirit of auxiliary tasks trained jointly with the main task (Situ et al.).

Data come from three sources. The first is Active TartanAir, re-rendered with ambient irradiance scaled over roughly two orders of magnitude relative to the pattern, using the illumination-dependent image formation that Polka Lines uses to model ambient washout. The second is the real indoor D435i Sequences. The third is a new capture: a D435-class sensor with its stock narrowband filter, logged with a lux meter from indoor shade to midday sun, with projector-on/off frames interleaved for decomposition supervision and a co-mounted LiDAR used only for evaluation. For an existing outdoor real-world test, we evaluate on M3ED-active, whose outdoor and night splits VPP used.

Baselines

The naive reference is ActiveStereoNet-style training: contrast-normalized input, active photometric loss only, invalidation mask. It is retrained on our exact data with the same encoder. It should excel indoors and collapse in sunlight. Any method that fails to beat it in the top ambient bins has failed. Any method that loses more than a small margin to it in the bottom bins has given up the reason active sensing exists.

The simple competitor is the same network with raw plus normalized input and a fixed equal-weight sum of active and passive losses. This tests whether giving the network both cues is already enough.

The strongest existing method is ACDC-Net's active/passive min-operator loss. We run it twice on the same backbone and data: once without SLAM landmarks, which isolates the loss design, and once with them, which reproduces the full system. Its paper reports gains over self-supervised and supervised baselines on Active TartanAir and D435i, but no reported result found on Lacuna for its RMSE, Rel, or δ values, so all comparisons use our re-runs.

For outdoor context, VPP reports reducing error rates by over 30% on M3ED outdoor splits relative to physical active stereo. Because it uses LiDAR seeds, we treat it as an upper reference that uses an extra sensor, not as a competitor.

The generic multi-task baseline applies RotoGrad to the two losses. The learned-weighting baseline replaces our physics gate with a per-pixel uncertainty predicted by the network and trained jointly with the losses. All methods get the same encoder, epochs, and a fixed hyperparameter sweep of 12 configurations. Ours sweeps $\alpha$ and $\tau$; the others sweep loss weights or learning rates.

Three ablations isolate our ingredients. An oracle gate computed from true on/off ratios bounds what better decomposition could buy. A gate predicted by a separate small network with no shared encoder tests whether sharing the decomposition task matters. Removing the stop-gradient tests the shortcut concern.

We count it as beating the field if two conditions hold. In the two highest ambient-to-pattern bins, bad-pixel rate drops relative to the min-operator baseline with non-overlapping seed intervals. In the lowest bin, end-point error stays within the naive baseline's seed variance.

Why not just…?

Why not just use a narrower bandpass filter? We already keep the sensor's narrowband infrared filter. The problem we study is what remains after it. The Dark Flash Normal Camera work explicitly notes that solar near-infrared overlaps active flash wavelengths. Structured Light in Sunlight shows that sunlight up to 90,000 lux still swamps spread-out projector power. Our lux-binned results report the residual washout directly.

Why not just concentrate the light, as in concentrate-and-scan? It works physically. It also scans sequentially, so it smears motion, it adds mechanical hardware, and the authors report that specular glints still leave holes. Our setting is solid-state cameras on moving platforms, where these costs rule it out. A concentrated pattern would simply move pixels into higher-$\hat\rho$ bins, so the two approaches are complementary.

Why not just train a single-task network on lots of mixed-illumination data and let it figure out which cue to use? This is the real threat. With dense synthetic ground truth, it may well suffice, and we will report that if it happens. Our argument is specific to self-supervised training on real data, where the active loss itself becomes the corrupting signal. The equal-weight baseline with both inputs tests exactly this. We also measure, for each ambient bin, the cosine similarity between the gradients of the active and passive losses on the shared encoder. If those gradients do not conflict in high-ambient pixels, our premise is wrong. In that case the simple baseline should match us, and we would conclude the gate is unnecessary.

Why not just let the network learn per-pixel uncertainty weights, or use ACDC-Net's minimum? Both are included as baselines. Our prediction is that both pick weights by loss magnitude. A washed-out pattern yields a flat, low-contrast reconstruction error, so it can look reliable. The physics gate does not have this failure mode, because it depends on an independently supervised quantity. The oracle-gate ablation and a plot of learned weight against true $\rho$ will show whether learned weights track the physics or drift from it.

Proposed Experiments

The main experiment trains every method on the ambient-scaled Active TartanAir data plus our real captures. It reports end-point error, bad-pixel rates at 1 and 3 pixels, and completeness (fraction of pixels with valid depth). Results are reported on Active TartanAir, D435i Sequences, our lux-logged captures against LiDAR, and M3ED-active outdoor and night splits. Every metric is stratified by the true ambient-to-pattern ratio, never averaged over whole images.

The decisive test uses frames that contain both shade and direct sun. Our core claim predicts that any global weighting, including RotoGrad, trades error in one region for the other. Per-pixel gating should improve both regions at once. We show this by sweeping a global weight and plotting the shaded-region error and sunlit-region error that each weight achieves, then checking whether our model lies below that trade-off curve.

A second check asks how much projector-off supervision is needed. We vary the fraction of training frames with on/off pairs from 1% to 50%, to determine whether occasional strobing, which commodity sensors can schedule, is enough. A third check compares the confidence head with ActiveStereoNet's invalidation mask on a filtering task: discard the least-confident pixels and plot error against the fraction of pixels kept. The question is whether $\hat\rho$-aware confidence keeps passive estimates that the mask would have thrown away.

If the gradient-conflict measurement shows no spatial structure, or the mixed-data single-task baseline matches us in the high-ambient bins, we will report that per-pixel routing is unnecessary for this sensor class.