Picking Robustness Per Pixel: Photometric Stereo Without the Clean-Surface Penalty

Photometric stereo recovers the 3D orientation of every surface point by photographing an object from one fixed camera under many different lights. The math is simplest for matte surfaces, where brightness is just albedo times the cosine between the surface normal and the light direction. Shiny highlights and shadows break that rule, so the field's standard tool is to treat them as outliers and reject them. Two ways of doing this are common: throw away the brightest and darkest measurements, or use loss functions that stop listening to large errors. Rejection has a cost that nobody measures. On a pixel that was already matte, rejecting data throws away good measurements and makes the estimate noisier. When only a handful of lights are available, rejection can leave too few usable light directions to determine a normal at all. The deep-learning survey of the field says this directly: networks built for shiny materials do worse on simple diffuse objects than plain least squares (Deep Learning Methods for Calibrated Photometric Stereo and Beyond). This is the photometric-stereo version of the robustness-accuracy trade-off known from adversarial classification, where defenses lower accuracy on clean inputs.

This note proposes a per-pixel rule that picks how much robustness to use. At every pixel we compute a ladder of estimates, from ordinary least squares (most efficient, least robust) up to heavily robust fits. We then accept the least robust estimate that is statistically consistent with every more robust one. Consistency is judged with each estimate's own noise covariance, and that covariance already reflects how well-spread the surviving light directions are. On matte pixels the ladder stops at least squares, so the clean-surface penalty should vanish by construction. On highlighted pixels it climbs to a robust fit. With sparse lights it cannot climb into a rank-deficient fit, because such a fit has an enormous covariance and so cannot disagree with anything.

Core Claim

The clean-accuracy loss of robust photometric stereo has two measurable parts. The first is variance inflation from down-weighting good measurements. The second is ill-conditioning from pruning light directions. Both can be predicted per pixel from the light matrix and the sensor noise level, without ground truth. We claim that a selection rule built on these predictions matches the best robust estimator on non-Lambertian objects on the DiLiGenT benchmark. We also claim it stays within noise of ordinary least squares on purely Lambertian surfaces rendered with the same lights, at both 96 and 10 lights. No fixed threshold or fixed loss function achieves both.

The robustness-accuracy literature supplies the lens here, not the method. Theoretically Principled Trade-off between Robustness and Accuracy splits robust error into natural error plus error near the decision boundary, and controls the split with one global weight λ tuned by retraining. Our split is the regression analogue: error on clean pixels versus bias from outliers that were not rejected. We differ from that work in one important way. We set the balance per pixel from observable statistics instead of with one global knob. That global knob is exactly what the file's open problem on "dynamic trade-off navigation" identifies as the limitation. The backdoor-cleansing results supply the diagnosis we borrow. Aggressive filters discard legitimate tail samples: Karami et al. report a 2–3% clean-accuracy drop from clean-data rejection (Training set cleansing of backdoor poisoning by self-supervised representation learning). In photometric stereo the tail samples are real grazing-angle and near-highlight diffuse measurements.

The Estimator

At a pixel with $m$ lights, stack the light directions into $L \in \mathbb{R}^{m\times 3}$ and the intensities into $I \in \mathbb{R}^m$. A weighted fit with weights $W=\mathrm{diag}(w)$ gives the scaled normal $b_W = (L^\top W L)^{-1} L^\top W I$. Under sensor noise of standard deviation $\sigma$, its covariance is

$$ \Sigma_W = \sigma^2 (L^\top W L)^{-1} L^\top W^2 L\, (L^\top W L)^{-1}. $$

This single expression carries both clean-accuracy costs. Down-weighting good rows inflates it (variance). Pruning toward coplanar lights makes $L^\top W L$ near-singular (conditioning).

The ladder $b_0, b_1, \dots, b_K$ is built in increasing order of robustness. The rungs are ordinary least squares, then Huber reweighting, then Cauchy and Geman-McClure reweighting at shrinking scales, then position thresholding at increasing discard fractions. Every rung includes the shadow-aware $\max(0, n\cdot s)$ term from A Non-convex Variational Approach to Photometric Stereo under Inaccurate Lighting. We accept rung $k$ if, for every more robust rung $j>k$,

$$ (b_k - b_j)^\top (\Sigma_k + \Sigma_j)^{-1} (b_k - b_j) \le \tau, $$

where $\tau$ is a chi-square quantile with 3 degrees of freedom. We output the smallest accepted $k$. This is Lepski-style adaptive estimator selection, applied per pixel. The test is conservative because it ignores the correlation between rungs; we will check its calibration on the Lambertian twins described below. We estimate $\sigma$ from the median absolute residual of the fit using the lights. Robust rungs whose pruned light matrix has condition number above a cap receive a very large covariance, so they never veto a less robust estimate.

This is a statistical fusion of existing estimators, not a new robust loss. The robust fits themselves are reused as they are. Photometric refinement of depth maps for multi-albedo objects supplies the Huber reweighting (iteratively reweighted least squares), and Quéau et al. supply the redescending losses solved by alternating reweighted least squares. As an optional extension, a pretrained photometric-stereo network can sit at the top of the ladder, with its covariance estimated by lighting-subset resampling. This lets the rule decide when to trust a network over least squares, which targets the survey's finding about diffuse objects directly.

Research Program

The first step is measurement, because the file contains no quantitative clean-surface penalty for any photometric-stereo method. For every DiLiGenT object we render a "Lambertian twin." It uses the laser-scanned ground-truth normals, the 96 calibrated light directions, a uniform albedo, attached shadows, and noise matched to the real captures. On a twin, ordinary least squares is the right estimator, so each method's excess error over least squares is a clean measurement of its penalty. We also construct "specular twins" by adding a controlled highlight lobe and cast shadows computed from the ground-truth geometry. The fraction of corrupted measurements is then known, which gives a controllable axis for robustness. Plotting twin error against real-object error for each method traces empirical trade-off curves.

The second step builds the selection rule described above. It also tests whether the variance predictions are accurate, by checking that the test's false-veto rate on Lambertian twins matches its nominal level.

The third step addresses a threat to the premise. Real specular lobes are spatially correlated and broad, as on the COW and BEAR objects. Several rungs could then share the same bias and agree with each other wrongly, so the test would accept a biased estimate. We will measure how often this happens on the specular twins as lobe width grows, and report it as a limitation if it proves significant. Translucent and strongly interreflecting surfaces violate the outlier model itself. We will report them separately rather than hide them, using the translucent class of DiLiGenT-II (DiLiGenT-Pi).

Baselines

The naive reference point is ordinary least squares with all lights. It defines zero clean penalty on the twins and gives the worst non-Lambertian error. All methods receive the same images, the same light calibration and the same shadow term.

The position-threshold baseline from A Benchmark Dataset and Evaluation for Non-Lambertian and Uncalibrated Photometric Stereo discards the top and bottom 40% of intensities and fits least squares. Shi et al. report that it rivals complex reflectance models when many images are available. The file quotes no overall MAE for it, so we will recompute it ourselves. A stronger variant chooses the discard fraction per light count by leave-one-object-out tuning on DiLiGenT, with the same tuning budget we give our own threshold $\tau$. That tuning is the honest version of this baseline.

The strongest classical competitor is redescending M-estimation with alternating reweighted least squares (Quéau et al.). They report 0.95° mean angular error on DiLiGenT Ball against 4.15° for their baseline. That result comes from their inaccurate-lighting setting, so we will rerun the method with calibrated lights and the same scale-tuning budget, and use it as our non-Lambertian target. For learned and inverse-rendering references, DANI-Net reports 6.54° overall DiLiGenT MAE. It is uncalibrated, which is a different setting, but its code is available (DANI-Net codebase), so we can measure its penalty on the twins. The survey reports that PX-Net reaches about 6.33° on Reading with 96 images, against 15.39° for its baseline. No PX-Net code appears in the file, so we will include per-pixel networks only if they are runnable. Otherwise we cite the number without claiming a head-to-head comparison.

Two ablations isolate the key ingredient. The first is a residual gate: use least squares if its residual passes a chi-square test, and the most robust rung otherwise. The second is a conditioning-only rule: use the most robust rung, but never prune below a condition-number cap. Beating the gate shows that the covariance-aware comparison between rungs matters. Beating the cap rule shows that variance inflation, and not only rank loss, matters.

We will count the proposal as a success under two conditions. First, it is within 0.2° of ordinary least squares on Lambertian twins at both 96 and 10 lights. Second, on real DiLiGenT it is no worse than 0.3° above the best-tuned robust baseline on every object and better on average at 10 lights. The 0.3° margin is our chosen tolerance, not a figure taken from the file.

Why not just…?

Why not just tune the discard fraction for each light count? A single global fraction still applies to matte and shiny pixels alike. Within one object, the pixels on the matte body therefore pay the variance cost that only the highlighted pixels justify. The tuned-threshold baseline is run exactly to measure this. The per-pixel penalty maps on the twins will show whether a global value can be close to per-pixel optimal. If it can, our method adds little, and we will say so.

Why not just use Huber loss everywhere? Huber loses little efficiency on clean data, so it is a genuine threat on the twins. Quéau et al. note, however, that its linear growth still lets strong highlights pull the normal. Huber is one rung of our ladder, and the comparison will show whether it alone sits on the trade-off frontier. If Huber dominates at both light counts, the ladder is unnecessary, and we will report that.

Why not just gate between least squares and a robust fit on the residual? This is the most serious alternative. A residual test ignores how much a robust fit's variance grows after pruning. With 10 lights, it can therefore switch to a fit that is rank-starved and worse than the biased least-squares fit. The gate ablation at 10 lights settles the question. If the gate matches us there, the contribution shrinks to the measurement protocol.

Why not just train a network on mixed matte and shiny data? The survey attributes its diffuse-object failures to overfitting to complex architectures and to specialized reflectance training distributions. DiLiGenT-Pi shows that fine-tuning moves shape biases around instead of removing them. Measuring networks on the Lambertian twins will quantify this penalty for the first time. Placing a network at the top of our ladder tests whether per-pixel selection recovers its clean accuracy without retraining.

Proposed Experiments

We will evaluate on all ten DiLiGenT objects at 96 lights and on 10-light subsets. The subsets are drawn repeatedly, with at least 20 draws per object, and we report medians. We will also evaluate on the DiLiGenT-II rough (near-matte), metallic and specular classes, and on the Lambertian and specular twins of each object. Metrics are mean angular error, clean penalty (excess twin error over ordinary least squares), and per-pixel error maps split by ground-truth shading regime.

Three checks decide the outcome. First, at 10 lights on the Lambertian twins, every fixed robust method must show a positive penalty and ours must not; otherwise the premise fails. Second, at 10 lights on real CAT (mostly diffuse) versus COW and READING (specular), our method must track the better of least squares and robust M-estimation on each object. Third, the gate ablation must lose at sparse lighting; otherwise the covariance machinery is not earning its place.