MantissaCam: Learning Snapshot High-dynamic-range Imaging with Perceptually-based In-pixel Irradiance Encoding

Haley M. So, Julien N. P. Martel, Piotr Dudek, Gordon Wetzstein

Abstract

The ability to image high-dynamic-range (HDR) scenes is crucial in many computer vision applications. The dynamic range of conventional sensors, however, is fundamentally limited by their well capacity, resulting in saturation of bright scene parts. To overcome this limitation, emerging sensors offer in-pixel processing capabilities to encode the incident irradiance. Among the most promising encoding schemes is modulo wrapping, which results in a computational photography problem where the HDR scene is computed by an irradiance unwrapping algorithm from the wrapped low-dynamic-range (LDR) sensor image. Here, we design a neural network–based algorithm that outperforms previous irradiance unwrapping methods and we design a perceptually inspired "mantissa" encoding scheme that more efficiently wraps an HDR scene into an LDR sensor. Combined with our reconstruction framework, MantissaCam achieves state-of-the-art results among modulo-type snapshot HDR imaging approaches. We demonstrate the efficacy of our method in simulation and show benefits of our algorithm on modulo images captured with a prototype implemented with a programmable sensor.

Introduction

Standard image sensors are limited by their "full well capacity"—once a pixel collects a certain amount of light, it saturates, turning bright highlights into flat white regions. Capturing the full range of light in natural scenes typically requires exposure bracketing (taking multiple photos at different speeds), which often introduces "ghosting" artifacts when motion is present. While other snapshot methods exist, they frequently sacrifice spatial resolution or require complex multi-sensor setups.

A promising alternative is the modulo camera, which resets a pixel's value to zero every time it hits its maximum capacity, effectively "wrapping" the irradiance. However, this creates a difficult unwrapping problem: reconstructing the original brightness from a wrapped image. Traditional modulo approaches use linear encoding, which requires numerous wraps for high-intensity areas, making reconstruction error-prone and wasting sensor precision on details the human eye cannot perceive.

Methodology

MantissaCam introduces two major innovations to solve the snapshot HDR problem:

  1. Mantissa Encoding: Instead of wrapping raw intensity linearly, the system wraps the logarithm of the intensity. This "mantissa" encoding is perceptually motivated by the Fechner-Weber law, which states that the human eye perceives light logarithmically. By allocating more sensor precision to dark areas and less to highlights, log-wrapping results in significantly fewer "wraps" for the same dynamic range, simplifying the reconstruction task.
  2. Neural Reconstruction Pipeline: A two-stage deep learning architecture was developed to unwrap the images. The first stage predicts "wrap edges"—the boundaries where the sensor rolled over. The second stage utilizes an Attention UNet to predict the "winding number" (the number of times each pixel wrapped) in a single pass. This direct prediction is more robust to sensor noise than previous iterative methods.
Overview of the MantissaCam pipeline, from in-pixel encoding to neural reconstruction.

Experimental Results

The researchers evaluated MantissaCam using both simulated data (593 HDR images) and real-world data captured with a prototype built using the SCAMP-5, a programmable focal-plane sensor-processor chip.

In simulations, MantissaCam achieved a PSNR of 37.4 dB, significantly outperforming the previous state-of-the-art, UnModNet (29.5 dB). On real-world data from the SCAMP-5 prototype, the system maintained high quality (33.7 dB PSNR), whereas previous methods often collapsed due to sensor noise at wrap boundaries. Furthermore, the method demonstrates excellent temporal consistency, producing stable HDR video from a single-shot sensor stream without the flickering common in prior modulo decoders.

Qualitative and quantitative comparison of MantissaCam against state-of-the-art methods across various scenes.

Discussion and Future Work

MantissaCam demonstrates that programmable vision chips can allow for "software-defined" sensing, enabling novel encoding schemes without requiring new hardware for every iteration. While the current prototype is grayscale and implemented programmatically on the SCAMP-5, future work includes developing dedicated CMOS sensors with built-in log-modulo hardware for even greater efficiency. Additionally, integrating Color Filter Arrays (CFA) and addressing noise in extremely dark shadows remain key areas for further development. While log-encoding reduces absolute precision at extremely high intensities, the authors argue this trade-off is optimal for human-centric viewing applications.

Summary updated

Abstract

The ability to image high-dynamic-range (HDR) scenes is crucial in many computer vision applications. The dynamic range of conventional sensors, however, is fundamentally limited by their well capacity, resulting in saturation of bright scene parts. To overcome this limitation, emerging sensors offer in-pixel processing capabilities to encode the incident irradiance. Among the most promising encoding schemes is modulo wrapping, which results in a computational photography problem where the HDR scene is computed by an irradiance unwrapping algorithm from the wrapped low-dynamic-range (LDR) sensor image. Here, we design a neural network--based algorithm that outperforms previous irradiance unwrapping methods and we design a perceptually inspired "mantissa" encoding scheme that more efficiently wraps an HDR scene into an LDR sensor. Combined with our reconstruction framework, MantissaCam achieves state-of-the-art results among modulo-type snapshot HDR imaging approaches. We demonstrate the efficacy of our method in simulation and show benefits of our algorithm on modulo images captured with a prototype implemented with a programmable sensor.
BibTeX
@inproceedings{so2022mantissacam,
  author = {Haley M. So and Julien N. P. Martel and Piotr Dudek and Gordon Wetzstein},
  title = {{MantissaCam: Learning Snapshot High-dynamic-range Imaging with Perceptually-based In-pixel Irradiance Encoding}},
  booktitle = {ICCP},
  year = {2022},
  doi = {10.1109/iccp54855.2022.9887659},
}
Figure