Learning Deep Illumination-Robust Features from Multispectral Filter Array Images

Anis AmzianeWACV 2025

TL;DR

This paper introduces a novel approach to extract robust features from multispectral (MS) images captured by snapshot cameras, bypassing the computationally expensive and artifact-prone demosaicing step. By learning directly from raw MSFA images and employing techniques like raw spectral constancy and MSFA-preserving augmentations, the proposed method achieves state-of-the-art performance in MS texture classification while being significantly more efficient.

What Problem Does This Paper Address?

Multispectral (MS) snapshot cameras, equipped with a Multispectral Filter Array (MSFA), capture multiple spectral bands in a single shot. However, each pixel in the resulting raw image only contains information from a single spectral band. To obtain a fully-defined MS image, a demosaicing process is required, which can introduce spatio-spectral artifacts and is computationally intensive, especially when using deep neural networks (DNNs). Furthermore, outdoor MS images are subject to varying illumination conditions, leading to features that are dependent on the lighting, hindering generalization. This paper addresses the challenge of learning discriminant and illumination-robust features directly from raw MSFA images, avoiding the pitfalls of demosaicing and illumination variations.

Methods

The authors propose a three-pronged approach:

  • Raw Spectral Constancy: They extend a statistical color constancy method to the raw image domain to mitigate the impact of illumination variations. This aims to make the raw pixel values more invariant to lighting changes.
  • MSFA-Preserving Augmentations: Traditional image augmentation techniques can disrupt the MSFA's basic pattern. The paper introduces novel augmentations (flips, translations, texture remodeling) that preserve the MSFA's structure, ensuring that the learned features are consistent across different spatial locations and augmentations. These augmentations are applied to the unshuffled raw image.
  • RawMixer Architecture: A hybrid deep learning model is proposed, combining a convolutional mixer and a transformer encoder. The initial convolutional layer is designed with a B-pixel stride to explicitly leverage the MSFA's basic pattern, capturing spatio-spectral interactions within raw image patches. This is followed by a transformer encoder for further feature learning.

Datasets

The experiments were conducted on two datasets:

  • HyTexiLa: This dataset contains 112 MS reflectance images, each representing a distinct class. The authors simulate radiance images under different illuminants (Solar, D65 simulator, extended A) and transform them into B²-channel images, then spatio-spectrally sub-sample them according to different MSFA configurations (4x4, 5x5, 2x2).
  • SpecTex: This dataset comprises 60 MS reflectance images of textures, each as a single class. Similar to HyTexiLa, radiance images are simulated and transformed into B²-channel images, then sub-sampled according to MSFA configurations.

Key Insights and Ideas

  • Learning from Raw is Superior: The paper strongly advocates for learning directly from raw MSFA images, demonstrating that it can yield comparable or even superior results to using demosaiced images, while being computationally more efficient.
  • Demosaicing is a Bottleneck: The demosaicing process, while necessary for traditional approaches, introduces artifacts that can hinder the learning of discriminative spatio-spectral features.
  • Illumination Invariance is Crucial: Directly addressing illumination variations through raw spectral constancy is key to achieving robust features that generalize across different lighting conditions.
  • Preserving MSFA Structure Matters: Augmentations that maintain the integrity of the MSFA basic pattern are essential for consistent spatio-spectral feature learning from raw images.

Key Results

  • The proposed RawMixer approach consistently outperforms both handcrafted and existing deep learning-based methods on MS texture classification tasks across various MSFA configurations and illuminants.
  • On the HyTexiLa dataset, RawMixer achieved the top rank in seven out of twelve tested cases.
  • On the SpecTex dataset, RawMixer ranked first in seven out of eight tested cases.
  • The method is significantly more computationally efficient, as it bypasses the demosaicing step. Figure 7 illustrates a favorable trade-off between accuracy and computation time for RawMixer compared to other methods.
  • Ablation studies confirm the critical importance of raw spectral constancy and MSFA-preserving augmentations for achieving illumination robustness and consistent feature learning.

Limitations and Future Work

  • Limitations: The paper acknowledges that while their approach is robust, the performance can still be impacted by the MSFA pattern size. The current work focuses on texture classification, and extending it to other tasks like segmentation is a natural next step.
  • Future Work: The authors suggest future research directions including:
  • Enhancing RawMixer by integrating raw spectral constancy directly into its architecture.
  • Developing a specialized encoder for MS image segmentation.
  • Exploring potential applications in outdoor crop/weed recognition.
  • Investigating the extension of their approach to the color domain by utilizing different Color Filter Arrays (CFAs).

Summary updated

BibTeX
@inproceedings{amziane2025learning,
  author = {Anis Amziane},
  title = {{Learning Deep Illumination-Robust Features from Multispectral Filter Array Images}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure