High Dynamic Range 3D Scene Reconstruction

13 concepts11 papers1 proposal

The real world exhibits an immense dynamic range of light, spanning from the deep shadows of a poorly lit room to the blinding radiance of the sun. Standard digital cameras, however, compress this vast spectrum into 8-bit Low Dynamic Range (LDR) images. This compression introduces non-linear camera response functions (CRFs) and truncates extreme values via clipping, turning bright highlights into uniform patches of pure white.

For 3D reconstruction and inverse rendering, this loss of information is catastrophic. If we want to realistically insert a virtual object into a captured scene or change its lighting, we need to know the true, linear physical radiance of the environment. A growing body of research addresses this challenge by merging 3D scene representations with physics-based radiance recovery, allowing models to reconstruct High Dynamic Range (HDR) environments from limited or compressed image data.

Operating in Linear RAW Space

The most direct way to preserve physical light values is to bypass the camera's lossy processing pipeline entirely. Standard JPEG images are the product of demosaicing, denoising, tone-mapping, and compression—steps that destroy the linear relationship between photon counts and pixel values. By training 3D representations directly on raw sensor data, models can operate in a space where light transport remains mathematically clean.

In NeRF in the Dark, Ben Mildenhall et al. demonstrated that optimizing a Neural Radiance Field directly on noisy, linear RAW data preserves the full dynamic range of the scene. Because raw sensor outputs are linearly proportional to scene radiance, the network can naturally handle extreme exposure variations and severe low-light noise.

While effective, optimizing neural radiance fields on raw data is computationally expensive. To make this pipeline practical for real-time applications, subsequent research transitioned this linear optimization to 3D Gaussian Splatting. Xin Jin et al. in Lighting Every Darkness with 3DGS and Kaushik Mitra et al. in HDRSplat showed that fitting explicit 3D Gaussians within a 14-bit linear HDR space yields rapid training and real-time rendering speeds. Crucially, maintaining a linear HDR representation within the 3D model enables flexible post-capture computational photography; users can retroactively adjust exposure, modify white balance, or apply custom tone-mapping curves to the rendered views with the same fidelity as a raw photograph.

Self-Calibrating the Camera Pipeline

When raw data is unavailable and researchers must work with standard LDR photographs, the challenge shifts to explicitly modeling and inverting the camera's internal pipeline. If a dataset contains images taken with varying exposures or white balances, the 3D reconstruction algorithm must distinguish between a change in the environment's light and a change in the camera's settings.

Kim Jun-Seong et al. addressed this in HDR-Plenoxels by incorporating a self-calibration mechanism directly into a voxel-based radiance field. By explicitly modeling the non-linear relationship between scene light and final pixel values, their framework can jointly optimize the 3D scene structure alongside camera parameters like exposure. This allows the system to synthesize novel views that reveal details hidden in deep shadows or saturated highlights.

A similar joint optimization strategy is vital for inverse rendering tasks, such as inserting virtual objects into real photographs. In Photorealistic Object Insertion with Diffusion-Guided Inverse Rendering, Ruofan Liang et al. jointly optimized HDR environmental lighting parameters and a spline-based tone-mapping curve. By accounting for the non-linear sensor response, the system can accurately estimate physical light intensities from standard LDR inputs, ensuring that inserted virtual objects exhibit consistent brightness and color.

Un-Clipping Highlights via Physical Light Transport

The most difficult problem in LDR-to-HDR reconstruction is recovering information that has been completely clipped to pure white. If a light source is saturated in every image, its pixels contain zero direct information about its true intensity. To recover the missing radiance, models must exploit the laws of physical light transport.

The core intuition is that the surrounding environment acts as a distributed sensor. While a light source itself might be clipped, the surfaces it illuminates are often not. Because light attenuates predictably as it travels and reflects off surfaces, the secondary illumination observed in the scene contains clues about the true intensity of the primary source.

[Saturated Light Source] (Clipped to White)
         │
         │  Light travels & attenuates
         ▼
[Unsaturated Surface] (Preserves Reflection Details)
         │
         │  Differentiable Path Tracing
         ▼
[Inferred True Radiance of Source]

In Pano-NeRF, Zhan Lu et al. leveraged this physical attenuation. By analyzing the reflected light on unsaturated surfaces and accounting for the scene's bidirectional reflection distribution function (BRDF), the model back-propagates errors from these surfaces to infer the true, unclipped intensity of the original light sources.

Chih-Hao Lin et al. formalized this concept for indoor environments in IRIS. The authors utilized differentiable path tracing to iteratively optimize the radiance of clipped light sources until the simulated global illumination matches the observed scene. This physics-based approach ensures that the restored HDR values are globally consistent with the entire environment, outperforming purely generative hallucination models.

A unique physical manifestation of this concept involves using natural neutral density filters found in the environment. In Appearance and Shape from Water Reflection, Ryo Kawahara et al. demonstrated that a water surface can act as a natural attenuator due to Fresnel reflection. Because the reflection dims the incoming light, the water surface provides an unclipped version of a scene that might be completely blown out in a direct view. Fusing the direct and reflected views allows for robust HDR reconstruction.

Cross-View and Prior-Driven Estimation

When explicit physical reflections are sparse or unavailable, models must rely on statistical, geometric, or cross-view priors to constrain the optimization.

For outdoor environments, the sky's appearance is tightly bound to the physical laws of atmospheric scattering. In Deep Outdoor Illumination Estimation, Yannick Hold-Geoffroy et al. trained a convolutional neural network to map standard LDR sky pixels to the parameters of a physically-based sky model. They demonstrated that the visible structure of the sky contains sufficient information to reconstruct the extreme, clipped luminance levels of the sun.

For complex 3D scenes captured from multiple angles, cross-view consistency serves as a powerful regularizer. Kaixuan Zhang et al. developed High Dynamic Range Novel View Synthesis with Single Exposure, a framework that reconstructs HDR 3D scenes from single-exposure images without requiring multi-exposure brackets. By enforcing consistency across different viewpoints and applying physical priors, the model successfully recovers details in both overexposed highlights and underexposed shadows.

Finally, when a complete 3D scene representation is already available but lacks dynamic range, external HDR data can be back-projected onto it. Manuele Sabbadin et al. showed in High Dynamic Range Point Clouds for Real-Time Relighting that a single uncalibrated HDR photograph can upgrade an entire LDR point cloud. By aligning exposures using Wasserstein distance and establishing spatial correspondences via patch matching, the framework back-projects the HDR values into the point cloud's overexposed regions, providing the high-fidelity radiance data required for realistic virtual object relighting.

Key Foundational Work

High Dynamic Range Point Clouds for Real-Time Relighting (2019) — contributed 2 concept elements to this direction.
Deep Outdoor Illumination Estimation (2017) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_joint_tone_mapping_and_lighting_optimiza_6446d4e5,
  title = {joint tone-mapping and lighting optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/joint-tone-mapping-and-lighting-optimization/rdr_6446d4e5e7584f50bcde149fcdb369a6},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_joint_tone_mapping_and_lighting_optimiza_6446d4e5_v1,
  title = {joint tone-mapping and lighting optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/joint-tone-mapping-and-lighting-optimization/rdr_6446d4e5e7584f50bcde149fcdb369a6/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake