Enhancing the Visible with the Invisble: RGB plus Near-Infrared

Clément Fredembach, Nathalie Barbuscia, Yue Lu, Neda Salamati, Lex Schaul, Daniel Tamburrino, Sabine SüsstrunkInfoscience 2010

Abstract

Silicon-based digital camera sensors exhibit significant sensitivity beyond the visible spectrum (400-700nm). They are able to capture wavelengths up to 1100 nm, i.e., they are sensitive to near-infrared (NIR) radiation. This additional information is conventionally treated as noise and is absorbed by a NIR-blocking filter affixed to the sensor. Retaining instead of removing NIR information can significantly improve certain computational photography and computer vision tasks. Indeed, intrinsic properties of the NIR wavelength band guarantee that images can be sharper, less affected by man-made colorants, and more resilient to changing light conditions. Exploiting the inherent correlation and de-correlation of the different waveband images can lead to improved color rendering and object recognition.

1. Introduction: The Problem of Discarded Data

Standard digital photography is designed to mimic human vision, which is limited to the 400–700nm range. To prevent "color pollution" and maintain perceptual accuracy, camera manufacturers typically place a near-infrared (NIR) blocking filter over silicon sensors. However, this ignores the fact that silicon is naturally sensitive up to 1100nm.

Silicon sensor sensitivity compared to human cone cells (S, M, L) across the electromagnetic spectrum.

This "invisible" NIR radiation contains valuable physical information that visible light lacks. Visible light is easily scattered by atmospheric particles (haze) and heavily absorbed by skin pigments like melanin and hemoglobin. By discarding NIR data, we lose a powerful tool for solving fundamental computer vision challenges like image restoration, scene understanding, and realistic editing.

2. Methodology

The research presents a suite of techniques to exploit the relationship between visible and NIR spectra:

2.1 Dehazing via WLS Optimization

Leveraging Rayleigh’s Law—which states that scattering is inversely proportional to the fourth power of the wavelength—the authors use the clearer NIR image as a guide. They employ a multi-resolution decomposition based on a Weighted Least Squares (WLS) framework to fuse the high-contrast details of the NIR image into the visible image.

2.2 Realistic Skin Smoothing

NIR light penetrates deeper into the epidermis and is less absorbed by surface pigments. The authors use a bilateral filter to decompose both visible and NIR images into "base" (low-frequency) and "detail" (high-frequency) layers. By fusing the smooth NIR detail with the visible base, they remove surface imperfections while maintaining a natural skin texture.

2.3 Shadow and Illuminant Detection

The researchers exploit the fact that natural materials typically have a higher reflectance in NIR than in RGB. By analyzing the ratios between color channels and the NIR signal, they can distinguish between actual shadows and dark-colored objects.

2.4 Joint Capture CFA Design

To move beyond laboratory setups requiring two separate cameras, the authors propose a novel Color Filter Array (CFA). This pattern contains 15 visible/NIR filters and one dedicated NIR-only filter, allowing a single sensor to capture both spectra simultaneously.

3. Results and Discussion

The integration of NIR data leads to several key performance breakthroughs:

  • Material Classification: Using only visible light features, classification accuracy was 60%. Adding NIR features increased this to 98%. This is driven by "NIR-Metamerism"—materials that look identical in the visible spectrum but appear completely different in NIR.
  • Image Dehazing: Qualitative results show significantly higher contrast and detail recovery in distant landscape features compared to state-of-the-art visible-only methods like the Dark Channel Prior.
Comparison of visible RGB (top-left), NIR (top-right), and fused dehazed image (bottom).
  • Shadow Detection: The proposed method successfully distinguishes between cast shadows and black objects (e.g., a black shirt), a common failure point for traditional algorithms.

4. Conclusion and Future Work

While the potential of NIR-RGB fusion is clear, hardware availability remains a primary limitation. Mass-market sensors with the proposed CFA technology do not yet exist. Furthermore, capturing four channels on a single mosaic increases the complexity of demosaicking, which can lead to spatial artifacts. Future research will focus on validating material classification on larger databases and refining color rendering to ensure fused images maintain a natural appearance.

Summary updated

Abstract

Silicon-based digital camera sensors exhibit significant sensitivity beyond the visible spectrum (400-700nm). They are able to capture wavelengths up to 1100 nm, i.e., they are sensitive to near-infrared (NIR) radiation. This additional information is conventionally treated as noise and is absorbed by a NIR-blocking filter affixed to the sensor. Retaining instead of removing NIR information can significantly improve certain computational photography and computer vision tasks. Indeed, intrinsic properties of the NIR wavelength band guarantee that images can be sharper, less affected by man-made colorants, and more resilient to changing light conditions. Exploiting the inherent correlation and de-correlation of the different waveband images can lead to improved color rendering and object recognition.
BibTeX
@inproceedings{fredembach2010enhancing,
  author = {Clément Fredembach and Nathalie Barbuscia and Yue Lu and Neda Salamati and Lex Schaul and Daniel Tamburrino and Sabine Süsstrunk},
  title = {{Enhancing the Visible with the Invisble: RGB plus Near-Infrared}},
  booktitle = {Infoscience},
  year = {2010},
}
Figure