When a camera pixel saturates, the usual practice is to throw it away. In fact the pixel still says something: the true brightness at that point was at least the sensor's maximum. The same reasoning applies to a white-light interferometer (an optical instrument that measures surface height to sub-nanometer precision). When it returns a "void," a hole with no height reading, that is also a measurement. It means the surface there tilted steeply enough to send light outside the objective lens. We propose to rebuild multi-modal 3D microstructure reconstruction around this idea. Both kinds of censored readings, saturated color pixels and interferometer voids, become one-sided constraints in a single fitting objective rather than gaps to fill or downweight. The pairing is natural because the two failures tend to happen in different places. Facets that face the microscope's coaxial light are the ones whose color pixels clip, and they are also the ones the interferometer measures best. Steep slopes that void the interferometer rarely produce glare in the camera.
The practical payoff is a reconstruction that is honest about what it does not know. Current fusion methods have no photometric signal at all on a facet that stays saturated in every view, so they output an unanchored average color without warning. Our method instead returns an interval of colors consistent with every bound, plus an explicit flag where the data cannot identify the color. In a metrology or materials setting, a confidently wrong texture is worse than a stated uncertainty.
Core Claim
OpticFusion supplies the base method. It fuses white-light interferometry with multi-view color microscopy in a neural signed-distance surface, a network that represents the surface as the zero level of a learned distance function. It splits color into a view-independent diffuse term and a view-dependent residual that absorbs LED glare. That residual is discarded at render time. On its synthetic benchmark it reaches a Chamfer distance of 0.043, against 0.538 for microscopy alone and 0.793 for interferometry alone.
The motivating gap comes from two papers. OpticFusion and Neural Exposure Fields, which learns a 3D exposure field and downweights clipped rays, both assume that some view of each point is unclipped. Neural Exposure Fields states outright that it cannot recover a region clipped in all views. Neither method treats clipping as a measurement.
The technique we borrow comes from Saturation in Snapshot Compressive Imaging, which reports gains of up to about 30 dB. Its method, SAPnet, enforces saturated measurements as lower-bound inequalities inside an iterative reconstruction. On the Drop video at a saturation ratio of 0.25 it beats a solver that ignores saturation, PnP-FastDVDnet, by 30.017 dB. That work is linear, two-dimensional and single-modality. We carry the inequality idea into a nonlinear, multi-view, two-instrument surface fit. We add a second, geometric kind of censoring (voids imply steepness), which has no analogue in compressive imaging.
Concretely, let $\hat y$ be the rendered color intensity for a camera ray, $y$ the recorded value, and $V$ the clipping level. Let $\theta(x)$ be the local surface tilt at point $x$ implied by the distance field, and $\theta_{NA}$ the acceptance half-angle of the interferometer objective. The photometric and void terms are
$$ \mathcal{L} = \sum_{y<V} (y-\hat y)^2 + \sum_{y=V} \max(0,\,V-\hat y)^2 + \lambda \sum_{x\in\text{voids}} \max(0,\,\theta_{NA}-\theta(x))^2 , $$
added to OpticFusion's existing height and regularization losses. Here $\lambda$ weights the void term. The first term is ordinary least squares on valid pixels. The second penalizes the model only if it renders a clipped pixel below the ceiling. The third penalizes the model only if it renders a voided point as flatter than the objective could have seen. The second and third terms are the one-sided analogue of a censored-data (Tobit) likelihood.
Research Program
The first component is the censored fusion objective above, implemented by modifying OpticFusion's NeuS-based (signed-distance neural rendering) pipeline. Everything else is kept fixed so that differences can be attributed to the loss.
The second component is identifiability analysis. On a fully clipped point, the interferometer gives the geometry, so the diffuse shading factor $s(x)$ is known. The clipping bound then gives a lower bound on the diffuse albedo, the surface's intrinsic color, once we subtract the largest specular contribution allowed by a specular lobe fitted on nearby unclipped pixels. The physical constraint that albedo is at most one gives the upper bound. We then add an optional material-segment prior: facets with matching interferometer-derived curvature and bordering unclipped color share albedo statistics. This prior picks a point estimate inside the interval. The prior's contribution is always reported separately from what the bounds alone imply. Points where the interval stays wide are flagged as non-identifiable rather than filled in.
The third component asks how much the void constraint helps geometry. Tilt bounds alone cannot fix the height inside a void. They can, however, rule out the smooth, flat interpolation that Poisson reconstruction and plain signed-distance smoothing tend to produce. We will measure where this matters, at steep sidewalls and seed ridges.
Baselines
The naive reference is OpticFusion with saturated pixels and voids simply masked out of the loss. This is the "discard clipped data" practice that SAPnet showed to be badly suboptimal in the compressive-imaging setting.
The strongest existing method is unmodified OpticFusion. It is the only published system on this exact two-instrument task, and its glare branch is designed to absorb clipping. It will be run from its released configuration on identical inputs, with the same iteration count and the same hyperparameter search budget as our variant. The reference figure is a Chamfer distance of 0.043 on its synthetic set. No reported result was found on Lacuna for color or albedo accuracy inside glare regions. We will therefore report OpticFusion's albedo error ourselves, split into unclipped, partially clipped and fully clipped surface regions.
A second competitor transplants Neural Exposure Fields' clipped-ray downweighting into the same backbone. This tests whether a soft version of masking is enough.
Two ablations isolate the key ingredient: the clipping inequality alone, and the void inequality alone. Beating the baselines means two things. Geometry must not get worse, so Chamfer distance must not rise above OpticFusion's on the original data. On the fully clipped regions, our intervals must contain the true albedo at their stated rate, while OpticFusion's point estimate misses by more than the interval width.
Why not just…?
Why not just bracket exposures? In optical microscopy, unlike electron or X-ray probes, extra exposures cause no beam damage. Bracketing is cheap and is the most serious threat to this proposal. If a short exposure unclips every facet, bracketing will recover color better than any bound. We treat it as an oracle rather than argue around it. Our real-data ground truth is a bracketed stack, and we will plot albedo error against the number of exposures for bracketing with plain merging versus bracketing with our censored loss. Two outcomes keep the proposal useful even if bracketing wins on color. First, the censored objective may reach bracketed accuracy with fewer exposures, which matters for long multi-view sequences and specimens that drift thermally. Second, the void constraint and the non-identifiability flags have no bracketing equivalent, since interferometer voids are geometric, not exposure, failures. The same exposure-sweep experiment also covers cross-polarized illumination where that hardware is available. Cross-polarization removes glare optically but costs light, and it does not help interferometer voids.
Why not just mask saturated pixels and inpaint them? Masking is our naive baseline. The SAPnet result, 30.017 dB better than a saturation-unaware solver on Drop, suggests that discarding the "at least $V$" information hurts. That evidence comes from linear compressive imaging, however, and may not transfer to multi-view rendering, where many unclipped views already constrain most points. The partial-clipping split in our evaluation will show exactly where the bound adds information beyond masking.
Why not just use a learned enhancement or generative prior? Exposure-correction networks such as CSEC, at 23.627 dB PSNR on the LCDP benchmark, and ECMamba, at 23.76 dB on the ME benchmark, recover detail only where some residual variance survives. The ECMamba authors themselves say that truly lost signal requires generative hallucination. A plausible but invented texture on a crystalline facet is the failure this proposal exists to prevent. We will nevertheless apply CSEC to each input view before fusion. We will then report both its error and how often its output falls outside our feasible interval, which is a direct measure of physically inconsistent hallucination.
Proposed Experiments
The synthetic study extends OpticFusion's simulated dataset. We render microstructures in high dynamic range with a fitted specular lobe and clip at a sweep of thresholds, so that the fraction of surface clipped in every view runs from near zero to a large share. Voids come from the existing slope-based interferometer simulation. The metrics are Chamfer distance; albedo error separately on unclipped, partially clipped and fully clipped regions; interval coverage, meaning the fraction of true albedos inside the predicted interval; and interval width.
The real-data study uses OpticFusion's specimens: flower seeds, butterfly wings and electronic micro-components. For each view we capture a single working exposure as input and a bracketed stack as held-out ground truth. We also compute the surface roughness parameters $S_a$ and $S_q$ to test whether the void constraint changes downstream metrology on steep sidewalls.
The decisive checks are as follows. On fully clipped regions, intervals must reach their nominal coverage while OpticFusion's errors exceed the interval width. Chamfer distance on void-heavy sidewalls must drop relative to masked fusion. If coverage fails, the specular-lobe extrapolation is wrong and the bound claim falls. If single-exposure censored fusion never approaches two-exposure bracketing on color, the practical case narrows to the void constraint and the uncertainty flags, and we will say so.
References
- OpticFusion: Multi-Modal Neural Implicit 3D Reconstruction of Microstructures by Fusing White Light Interferometry and Optical Microscopy
- Learning Neural Exposure Fields for View Synthesis
- Saturation in Snapshot Compressive Imaging
- Color Shift Estimation-and-Correction for Image Enhancement
- ECMamba: Consolidating Selective State Space Model with Retinex Guidance for Efficient Multiple Exposure Correction