Incorporating Near-Infrared Information into Semantic Image Segmentation

Neda Salamati, Diane Larlus, Gabriela Csurka, Sabine SüsstrunkarXiv 2014 · arXiv

Introduction

Semantic image segmentation—the task of assigning a category label to every pixel in an image—is a cornerstone of computer vision. However, relying solely on visible light (RGB) data presents significant challenges. Objects of the same class can have wildly different colors, while different materials can look identical to the human eye (e.g., green paint vs. green grass). Furthermore, environmental factors like atmospheric haze can blur boundaries, and cluttered backgrounds often lead to "wrong edges" that confuse standard algorithms.

This research demonstrates that adding a single near-infrared (NIR) channel to standard RGB images significantly boosts segmentation accuracy. By leveraging material-specific NIR reflectance, the proposed framework better distinguishes between classes that appear ambiguous in the visible spectrum.

Challenging cases where RGB-only segmentation often fails: (a) different colors, same material; (b) haze; (c) texture vs. material.

Methodology

The authors employ a state-of-the-art Conditional Random Field (CRF) framework consisting of two main components:

  1. The Unary Term (Recognition): This component predicts the most likely label for a pixel based on local features. The researchers used Fisher Vectors (FV) to encode texture (via SIFT descriptors) and color statistics. They introduced a novel multi-spectral SIFT (SIFT_rgbn) that incorporates gradients from all four channels (R, G, B, and NIR).
  2. The Pairwise Term (Regularization): This component encourages neighboring pixels to share the same label unless a strong edge is detected. Crucially, the authors used the NIR channel to guide these boundaries, as NIR is often more sensitive to changes in material than visible light.

Datasets

To evaluate their approach, the authors introduced a novel manually segmented database of 770 registered RGB+NIR image pairs:

  • Outdoor Dataset: 370 images with 10 classes (e.g., building, grass, sky, tree, water).
  • Indoor Dataset: 400 images featuring 13 object categories common in office environments (e.g., screen, keyboard, mouse, bottle).
Sample images from the outdoor dataset showing RGB, NIR, and the groundtruth segmentation.

Key Insights

  • Material over Color: NIR reflectance is determined by the chemical and physical structure of a material, not just its surface dye. This allows the model to identify materials regardless of their visible color.
  • The "Chlorophyll Beacon": Vegetation is exceptionally bright in NIR due to chlorophyll, while water is highly absorbent (appearing very dark). This makes natural scene segmentation much more robust.
  • Haze Transparency: Atmospheric haze is largely transparent to NIR, allowing the model to "see through" fog to find sharp edges of distant objects that are washed out in RGB.
  • Indoor "Color Distractors": Computer screens are major distractors in RGB because their content changes constantly. In NIR, screens appear consistently dark and uniform, helping the model ignore the "noise" of the displayed image.

Results and Conclusion

Incorporating NIR led to consistent quantitative improvements. In outdoor scenes, Overall Accuracy (OA) rose from a visible baseline of ~83.5% to 86.3%. In more challenging indoor scenes, OA improved significantly from ~56.5% to 68.0%.

Qualitatively, the NIR-guided CRF produced much sharper and more accurate object boundaries. The multi-spectral SIFT (SIFT_rgbn) was the top performer, proving that explicitly modeling gradients across both visible and infrared light is superior to simple channel concatenation. This work highlights NIR as a powerful "physical signature" that can resolve fundamental ambiguities in computer vision.

Summary updated

Abstract

Recent progress in computational photography has shown that we can acquire near-infrared (NIR) information in addition to the normal visible (RGB) band, with only slight modifications to standard digital cameras. Due to the proximity of the NIR band to visible radiation, NIR images share many properties with visible images. However, as a result of the material dependent reflection in the NIR part of the spectrum, such images reveal different characteristics of the scene. We investigate how to effectively exploit these differences to improve performance on the semantic image segmentation task. Based on a state-of-the-art segmentation framework and a novel manually segmented image database (both indoor and outdoor scenes) that contain 4-channel images (RGB+NIR), we study how to best incorporate the specific characteristics of the NIR response. We show that adding NIR leads to improved performance for classes that correspond to a specific type of material in both outdoor and indoor scenes. We also discuss the results with respect to the physical properties of the NIR response.
BibTeX
@misc{salamati2014incorporating,
  author = {Neda Salamati and Diane Larlus and Gabriela Csurka and Sabine Süsstrunk},
  title = {{Incorporating Near-Infrared Information into Semantic Image Segmentation}},
  year = {2014},
  eprint = {1406.6147},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1406.6147},
}
Figure