Lumosaic: Hyperspectral Video via Active Illumination and Coded-Exposure Pixels

Dhruv Verma, Andrew Qiu, Roberto Rangel, Ayandev Barman, Hao Yang, Chenjia Hu, Fengqi Zhang, Roman Genov, David B. Lindell, Kiriakos N. Kutulakos, Alex Mariakakis

Abstract

We present Lumosaic, a compact active hyperspectral video system designed for real-time capture of dynamic scenes. Our approach combines a narrowband LED array with a coded-exposure-pixel (CEP) camera capable of high-speed, per-pixel exposure control, enabling joint encoding of scene information across space, time, and wavelength within each video frame. Unlike passive snapshot systems that divide light across multiple spectral channels simultaneously and assume no motion during a frame's exposure, Lumosaic actively synchronizes illumination and pixel-wise exposure, improving photon utilization and preserving spectral fidelity under motion. A learning-based reconstruction pipeline then recovers 31-channel hyperspectral (400-700 nm) video at 30 fps and VGA resolution, producing temporally coherent and spectrally accurate reconstructions. Experiments on synthetic and real data demonstrate that Lumosaic significantly improves reconstruction fidelity and temporal stability over existing snapshot hyperspectral imaging systems, enabling robust hyperspectral video across diverse materials and motion conditions.

Introduction

Hyperspectral imaging (HSI) captures the "fingerprint" of materials by measuring reflectance across many narrow wavelength bands, far beyond the standard Red, Green, and Blue. However, doing this in real-time for moving scenes is notoriously difficult. Traditional HSI systems usually fall into two categories: scanning systems, which are high-fidelity but slow and prone to motion blur, and snapshot systems, which use complex optics to capture everything at once but often suffer from "spatial-spectral mixing" and bulky form factors. Lumosaic aims to bridge this gap, providing a compact, "all-silicon" solution that is robust to rapid motion and efficient enough for low-light video.

System Design and Methods

The Lumosaic system relies on a co-design of hardware and software to achieve high-resolution spectral video. The hardware consists of two primary components: an active illumination module and a specialized sensor.

The Lumosaic hardware prototype (A) and the spatio-spectro-temporal coding scheme (B) that synchronizes LED illumination with pixel-wise exposure.
  • Active Illumination: Instead of filtering white light, Lumosaic uses an array of 12 high-power, narrowband LEDs. These LEDs cycle rapidly through different wavelengths within a single 30fps video frame.
  • Coded-Exposure-Pixel (CEP) Camera: This sensor allows for per-pixel exposure control at microsecond scales. Lumosaic programs a time-varying mosaic pattern that dictates exactly which pixels are active during specific LED pulses.
  • Temporal Alignment (RIFE): To account for motion between sub-frame captures, an optical flow network (RIFE) warps sub-images to a common reference time (the "Lime" LED pulse) before reconstruction.
  • Learning-Based Reconstruction: The aligned sub-images are processed by a modified Holistic Attention Network (HAN). This deep learning model is trained to "unshuffle" the coded measurements and reconstruct a full 31-channel hyperspectral cube (400–700 nm) at VGA resolution.

Key Results

Lumosaic consistently outperformed state-of-the-art snapshot methods across metrics such as Peak Signal-to-Noise Ratio (PSNR) and Spectral Angle Mapper (SAM). The system's active approach allows each LED’s full output to contribute to the signal, significantly improving the signal-to-noise ratio compared to passive systems that divide light across channels.

Real-world capture of a rotating globe (A, B) and the resulting 31-channel spectral reconstruction (C) from 400 nm to 700 nm.

In real-world evaluations, the system successfully reconstructed 30 fps video of rapidly moving objects, such as a rotating globe and diffusing liquids, with minimal ghosting. Furthermore, Lumosaic demonstrated the ability to disambiguate metamers—materials that appear identical to the human eye but have distinct spectral signatures, such as real paint versus printer ink.

Conclusion and Future Work

Lumosaic represents a significant step toward practical, high-speed hyperspectral video. By moving spectral encoding from bulky optics to the LEDs and the sensor itself, the system becomes compact and robust. Future work will explore utilizing inter-frame information to further stabilize video, addressing the scarcity of high-quality hyperspectral video datasets, and implementing adaptive coding patterns that change based on scene content to further optimize reconstruction fidelity.

Summary updated

Abstract

We present Lumosaic, a compact active hyperspectral video system designed for real-time capture of dynamic scenes. Our approach combines a narrowband LED array with a coded-exposure-pixel (CEP) camera capable of high-speed, per-pixel exposure control, enabling joint encoding of scene information across space, time, and wavelength within each video frame. Unlike passive snapshot systems that divide light across multiple spectral channels simultaneously and assume no motion during a frame's exposure, Lumosaic actively synchronizes illumination and pixel-wise exposure, improving photon utilization and preserving spectral fidelity under motion. A learning-based reconstruction pipeline then recovers 31-channel hyperspectral (400-700 nm) video at 30 fps and VGA resolution, producing temporally coherent and spectrally accurate reconstructions. Experiments on synthetic and real data demonstrate that Lumosaic significantly improves reconstruction fidelity and temporal stability over existing snapshot hyperspectral imaging systems, enabling robust hyperspectral video across diverse materials and motion conditions.
BibTeX
@inproceedings{verma2026lumosaic,
  author = {Dhruv Verma and Andrew Qiu and Roberto Rangel and Ayandev Barman and Hao Yang and Chenjia Hu and Fengqi Zhang and Roman Genov and David B. Lindell and Kiriakos N. Kutulakos and Alex Mariakakis},
  title = {{Lumosaic: Hyperspectral Video via Active Illumination and Coded-Exposure Pixels}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2026},
  url = {https://openreview.net/forum?id=jytZfaUfoC},
}
Figure