Abstract
We present Lumosaic, a compact active hyperspectral video system designed for real-time capture of dynamic scenes. Our approach combines a narrowband LED array with a coded-exposure-pixel (CEP) camera capable of high-speed, per-pixel exposure control, enabling joint encoding of scene information across space, time, and wavelength within each video frame. Unlike passive snapshot systems that divide light across multiple spectral channels simultaneously and assume no motion during a frame's exposure, Lumosaic actively synchronizes illumination and pixel-wise exposure, improving photon utilization and preserving spectral fidelity under motion. A learning-based reconstruction pipeline then recovers 31-channel hyperspectral (400-700 nm) video at 30 fps and VGA resolution, producing temporally coherent and spectrally accurate reconstructions. Experiments on synthetic and real data demonstrate that Lumosaic significantly improves reconstruction fidelity and temporal stability over existing snapshot hyperspectral imaging systems, enabling robust hyperspectral video across diverse materials and motion conditions.
Introduction
Hyperspectral imaging (HSI) captures the "fingerprint" of materials by measuring reflectance across many narrow wavelength bands, far beyond the standard Red, Green, and Blue. However, doing this in real-time for moving scenes is notoriously difficult. Traditional HSI systems usually fall into two categories: scanning systems, which are high-fidelity but slow and prone to motion blur, and snapshot systems, which use complex optics to capture everything at once but often suffer from "spatial-spectral mixing" and bulky form factors. Lumosaic aims to bridge this gap, providing a compact, "all-silicon" solution that is robust to rapid motion and efficient enough for low-light video.
System Design and Methods
The Lumosaic system relies on a co-design of hardware and software to achieve high-resolution spectral video. The hardware consists of two primary components: an active illumination module and a specialized sensor.
- Active Illumination: Instead of filtering white light, Lumosaic uses an array of 12 high-power, narrowband LEDs. These LEDs cycle rapidly through different wavelengths within a single 30fps video frame.
- Coded-Exposure-Pixel (CEP) Camera: This sensor allows for per-pixel exposure control at microsecond scales. Lumosaic programs a time-varying mosaic pattern that dictates exactly which pixels are active during specific LED pulses.
- Temporal Alignment (RIFE): To account for motion between sub-frame captures, an optical flow network (RIFE) warps sub-images to a common reference time (the "Lime" LED pulse) before reconstruction.
- Learning-Based Reconstruction: The aligned sub-images are processed by a modified Holistic Attention Network (HAN). This deep learning model is trained to "unshuffle" the coded measurements and reconstruct a full 31-channel hyperspectral cube (400–700 nm) at VGA resolution.
Key Results
Lumosaic consistently outperformed state-of-the-art snapshot methods across metrics such as Peak Signal-to-Noise Ratio (PSNR) and Spectral Angle Mapper (SAM). The system's active approach allows each LED’s full output to contribute to the signal, significantly improving the signal-to-noise ratio compared to passive systems that divide light across channels.
In real-world evaluations, the system successfully reconstructed 30 fps video of rapidly moving objects, such as a rotating globe and diffusing liquids, with minimal ghosting. Furthermore, Lumosaic demonstrated the ability to disambiguate metamers—materials that appear identical to the human eye but have distinct spectral signatures, such as real paint versus printer ink.
Conclusion and Future Work
Lumosaic represents a significant step toward practical, high-speed hyperspectral video. By moving spectral encoding from bulky optics to the LEDs and the sensor itself, the system becomes compact and robust. Future work will explore utilizing inter-frame information to further stabilize video, addressing the scarcity of high-quality hyperspectral video datasets, and implementing adaptive coding patterns that change based on scene content to further optimize reconstruction fidelity.
Summary updated